Amazon AWS EKS and RDS PostgreSQL with terraform

This is the second part of the 3 part series article on how to use terraform to deploy on Cloud providers Kubernetes offerings. In previous Article I showed how you can deploy complete kubernetes setup on Google Cloud GKE and PostgreSQL google sql offering. In this Article I will show how can you deploy Amazon AWS EKS and RDS with terraform. As AWS EKS is most recent service Amazon AWS cloud provider that adopted EKS Managed Kubernetes, be aware about the additional cost of $0.20 per hour for the EKS Control Plane “Kubernetes Master”, and usual EC2, EBS,etc prices for resources that run in your account. As comparing EKS with GKE is not so strait forward to deploy and configure requires more moving pieces like setting up AWS launch configuration and AWS autoscaling group and in addition IAM roles and policy to allow AWS to manage EKS.

NOTE: This tutorial is not secured and is not production ready

Article is structured in 5 parts

• Initial tooling setup aws cli , kubectl and terraform
• Creating terraform IAM account with access keys and access policy
• Creating back-end storage for tfstate file in AWS S3
• Creating Kubernetes cluster on AWS EKS and RDS on PostgreSQL
• Working with kubernetes “kubectl” in EKS

Initial tooling setup aws-cli, kubectl, terraform and aws-iam-authenticator

Assuming you already have AWS account and AWS CLI installed and AWS CLI configured for your user account we will need additional binaries for, terraform and kubectl.

Deploying terraform

terraform for OS X

curl -o terraform_0.11.7_darwin_amd64.zip https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_darwin_amd64.zip
unzip terraform_0.11.7_linux_amd64.zip -d /usr/local/bin/

terraform for Linux

curl https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip > terraform_0.11.7_linux_amd64.zip
unzip terraform_0.11.7_linux_amd64.zip -d /usr/local/bin/

terraform installation verification

Verify terraform version 0.11.7 or higher is installed

terraform version

Deploying kubectl

kubectl for OS X

curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/darwin/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

kubectl for Linux

wget https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

kubectl installation verification

kubectl version --client

Deploying aws-iam-authenticator

aws-iam-authenticator is a tool developed by Heptio Team and this tool will allow us to manage EKSby using kubectl

aws-iam-authenticator for OS X

curl -o aws-iam-authenticator \
https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/darwin/amd64/aws-iam-authenticator
chmod +x ./aws-iam-authenticator
cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$HOME/bin:$PATH

aws-iam-authenticator for Linux

curl -o aws-iam-authenticator \
https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator
chmod +x ./aws-iam-authenticator
cp ./aws-iam-authenticator $HOME/.local/bin/aws-iam-authenticator && export PATH=$HOME/bin:$PATH

aws-iam-authenticator installation verification

aws-iam-authenticator help

Authenticate to AWS

Before configuring AWS CLI as EKS at this time is only available in US East (N. Virginia) and US West (Oregon) In below example we will be using US West (Oregon) “us-west-2”

aws configure

Creating terraform IAM account with access keys and access policy

1st step is to setup terraform admin account in AWS IAM

Create IAM terraform User

aws iam create-user --user-name terraform

Add to newly created terraform user IAM admin policy

NOTE: For production or event proper testing account you may need tighten up and restrict access for terraform IAM user

aws iam attach-user-policy --user-name \
terraform --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

Create access keys for the user

NOTE: This Access Key and Secret Access Key will be used by terraform to manage infrastructure deployment

aws iam create-access-key --user-name terraform

Update terraform.tfvars file with access and security keys for newly created terraform IAM account

Creating back-end storage for tfstate file in AWS S3

Once we have terraform IAM account created we can proceed to next step creating dedicated bucket to keep terraform state files

Create terraform state bucket

NOTE: Change name of the bucket, name should be unique across all AWS S3 buckets

aws s3 mb s3://terra-state-bucket --region us-west-2

Enable versioning on the newly created bucket

aws s3api put-bucket-versioning --bucket \
terra-state-bucket --versioning-configuration Status=Enabled

Creating Kubernetes cluster on AWS EKS and RDS on PostgreSQL

Now we can move into creating new infrastructure, EKS and RDS with terraform

.
├── backend.tf
├── eks
│   ├── eks_cluster
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── eks_iam_roles
│   │   ├── main.tf
│   │   └── outputs.tf
│   ├── eks_node
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── userdata.tpl
│   │   └── variables.tf
│   └── eks_sec_group
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── main.tf
├── network
│   ├── route
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── sec_group
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── subnets
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   └── vpc
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── outputs.tf
├── rds
│   ├── main.tf
│   ├── outputs.tf
│   └── variables.tf
├── README.md
├── terraform.tfvars
├── variables.tf
└── yaml
    ├── eks-admin-cluster-role-binding.yaml
    └── eks-admin-service-account.yaml

We will use terraform modules to keep our code clean and organized Terraform will run 2 separate environment dev and prod using same sources only difference in this case is number of worker nodes for kubernetes.

Terraform modules will create

• VPC
• Subnets
• Routes
• IAM Roles for master and nodes
• Security Groups “Firewall” to allow master and nodes to communicate
• EKS cluster
• Autoscaling Group will create nodes to be added to the cluster
• Security group for RDS
• RDS with PostgreSQL

NOTE: very important to keep tags as if tags is not specify nodes will not be able to join cluster

Initial setup create and create new workspace for terraform

cd into project folder and create workspace for dev and prod

Initialize and pull terraform cloud specific dependencies

terraform init

Create dev workspace

terraform workspace new dev

List available workspace

terraform workspace list

Select dev workspace

terraform workspace select dev

Before we can start will need to update variables and add db password to terraform.tfvars

echo 'db_password = "Your_DB_Passwd."' >> terraform.tfvars

It’s a good idea to sync terraform modules

terraform get -update

View terraform plan

terraform plan

Apply terraform plan

NOTE: building complete infrastructure may take more than 10 minutes.

terraform apply

Verify instance creation

aws ec2 describe-instances --output table

We are not done yet

Create new AWS CLI profile

In order to use kubectl with EKS we need to set new AWS CLI profile

NOTE: will need to use secret and access keys from terraform.tfvars

cat terraform.tfvars
aws configure --profile terraform
export AWS_PROFILE=terraform

Configure kubectl to allow us to connect to EKS cluster

In terraform configuration we output configuration file for kubectl

terraform output kubeconfig

Add output of “terraform output kubeconfig” to ~/.kube/config-devel

terraform output kubeconfig > ~/.kube/config-devel
export KUBECONFIG=$KUBECONFIG:~/.kube/config-devel

Verify kubectl connectivity

kubectl get namespaces
kubectl get services

Second part we need to allow EKS to add nodes by running configmap

terraform output config_map_aws_auth > yaml/config_map_aws_auth.yaml
kubectl apply -f yaml/config_map_aws_auth.yaml

Now you should be able to see nodes

kubectl get nodes

Working with terraform on EKS

Deploy the Kubernetes Dashboard

Deploy the Kubernetes dashboard

kubectl apply -f \
https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

Deploy heapster to enable container cluster monitoring and performance analysis on your cluster

kubectl apply -f \
https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml

Deploy the influxdb backend for heapster to your cluster

kubectl apply -f \
https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml

Create the heapster cluster role binding for the dashboard

kubectl apply -f \
https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml

Create an eks-admin Service Account and Cluster Role Binding

Apply the service account to your cluster

kubectl apply -f yaml/eks-admin-service-account.yaml

Apply the cluster role binding to your cluster

kubectl apply -f yaml/eks-admin-cluster-role-binding.yaml

Connect to the Dashboard

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}')
kubectl proxy

NOTE: Open the link with a web browser to access the dashboard endpoint: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

NOTE: Choose Token and paste output from the previous command into the Token field

Rolling back all changes

Destroy all terraform created infrastructure

terraform destroy -auto-approve

Removing S3 bucket, IAM roles and terraform account

export AWS_PROFILE=default
aws s3 rm s3://terra-state-bucket --recursive
aws s3api put-bucket-versioning --bucket terra-state-bucket --versioning-configuration Status=Suspended
aws s3api delete-objects --bucket terra-state-bucket --delete \
"$(aws s3api list-object-versions --bucket terra-state-bucket | \
jq '{Objects: [.Versions[] | {Key:.Key, VersionId : .VersionId}], Quiet: false}')"
aws s3 rb s3://terra-state-bucket --force
aws iam detach-user-policy --user-name terraform --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
aws iam list-access-keys --user-name terraform  --query 'AccessKeyMetadata[*].{ID:AccessKeyId}' --output text
aws iam delete-access-key --user-name terraform --access-key-id OUT_KEY
aws iam delete-user --user-name terraform

Terraform and kubernetes sources can be found in GitHub