Amazon AWS EKS and RDS PostgreSQL with terraform
This is the second part of the 3 part series article on how to use terraform to deploy on Cloud providers Kubernetes offerings. In previous Article I showed how you can deploy complete kubernetes setup on Google Cloud GKE and PostgreSQL google sql offering. In this Article I will show how can you deploy Amazon AWS EKS and RDS with terraform. As AWS EKS is most recent service Amazon AWS cloud provider that adopted EKS Managed Kubernetes, be aware about the additional cost of $0.20 per hour for the EKS Control Plane “Kubernetes Master”, and usual EC2, EBS,etc prices for resources that run in your account. As comparing EKS with GKE is not so strait forward to deploy and configure requires more moving pieces like setting up AWS launch configuration and AWS autoscaling group and in addition IAM roles and policy to allow AWS to manage EKS.
NOTE: This tutorial is not secured and is not production ready
Article is structured in 5 parts
- Initial tooling setup aws cli , kubectl and terraform
- Creating terraform IAM account with access keys and access policy
- Creating back-end storage for tfstate file in AWS S3
- Creating Kubernetes cluster on AWS EKS and RDS on PostgreSQL
- Working with kubernetes “kubectl” in EKS
Initial tooling setup aws-cli, kubectl, terraform and aws-iam-authenticator
Assuming you already have AWS account and AWS CLI installed and AWS CLI configured for your user account we will need additional binaries for, terraform and kubectl.
Deploying terraform
terraform for OS X
curl -o terraform_0.11.7_darwin_amd64.zip https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_darwin_amd64.zipunzip terraform_0.11.7_linux_amd64.zip -d /usr/local/bin/
terraform for Linux
curl https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip > terraform_0.11.7_linux_amd64.zipunzip terraform_0.11.7_linux_amd64.zip -d /usr/local/bin/
terraform installation verification
Verify terraform version 0.11.7 or higher is installed
terraform version
Deploying kubectl
kubectl for OS X
curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/darwin/amd64/kubectlchmod +x kubectlsudo mv kubectl /usr/local/bin/
kubectl for Linux
wget https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/linux/amd64/kubectlchmod +x kubectlsudo mv kubectl /usr/local/bin/
kubectl installation verification
kubectl version --client
Deploying aws-iam-authenticator
aws-iam-authenticator is a tool developed by Heptio Team and this tool will allow us to manage EKSby using kubectl
aws-iam-authenticator for OS X
curl -o aws-iam-authenticator \
https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/darwin/amd64/aws-iam-authenticatorchmod +x ./aws-iam-authenticatorcp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$HOME/bin:$PATH
aws-iam-authenticator for Linux
curl -o aws-iam-authenticator \
https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticatorchmod +x ./aws-iam-authenticatorcp ./aws-iam-authenticator $HOME/.local/bin/aws-iam-authenticator && export PATH=$HOME/bin:$PATH
aws-iam-authenticator installation verification
aws-iam-authenticator help
Authenticate to AWS
Before configuring AWS CLI as EKS at this time is only available in US East (N. Virginia) and US West (Oregon) In below example we will be using US West (Oregon) “us-west-2”
aws configure
Creating terraform IAM account with access keys and access policy
1st step is to setup terraform admin account in AWS IAM
Create IAM terraform User
aws iam create-user --user-name terraform
Add to newly created terraform user IAM admin policy
NOTE: For production or event proper testing account you may need tighten up and restrict access for terraform IAM user
aws iam attach-user-policy --user-name \
terraform --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
Create access keys for the user
NOTE: This Access Key and Secret Access Key will be used by terraform to manage infrastructure deployment
aws iam create-access-key --user-name terraform
Update terraform.tfvars file with access and security keys for newly created terraform IAM account
Creating back-end storage for tfstate file in AWS S3
Once we have terraform IAM account created we can proceed to next step creating dedicated bucket to keep terraform state files
Create terraform state bucket
NOTE: Change name of the bucket, name should be unique across all AWS S3 buckets
aws s3 mb s3://terra-state-bucket --region us-west-2
Enable versioning on the newly created bucket
aws s3api put-bucket-versioning --bucket \
terra-state-bucket --versioning-configuration Status=Enabled
Creating Kubernetes cluster on AWS EKS and RDS on PostgreSQL
Now we can move into creating new infrastructure, EKS and RDS with terraform
.
├── backend.tf
├── eks
│ ├── eks_cluster
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ ├── eks_iam_roles
│ │ ├── main.tf
│ │ └── outputs.tf
│ ├── eks_node
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ ├── userdata.tpl
│ │ └── variables.tf
│ └── eks_sec_group
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
├── main.tf
├── network
│ ├── route
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ ├── sec_group
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ ├── subnets
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ └── vpc
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
├── outputs.tf
├── rds
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
├── README.md
├── terraform.tfvars
├── variables.tf
└── yaml
├── eks-admin-cluster-role-binding.yaml
└── eks-admin-service-account.yaml
We will use terraform modules to keep our code clean and organized Terraform will run 2 separate environment dev and prod using same sources only difference in this case is number of worker nodes for kubernetes.
Terraform modules will create
- VPC
- Subnets
- Routes
- IAM Roles for master and nodes
- Security Groups “Firewall” to allow master and nodes to communicate
- EKS cluster
- Autoscaling Group will create nodes to be added to the cluster
- Security group for RDS
- RDS with PostgreSQL
NOTE: very important to keep tags as if tags is not specify nodes will not be able to join cluster
Initial setup create and create new workspace for terraform
cd into project folder and create workspace for dev and prod
Initialize and pull terraform cloud specific dependencies
terraform init
Create dev workspace
terraform workspace new dev
List available workspace
terraform workspace list
Select dev workspace
terraform workspace select dev
Before we can start will need to update variables and add db password to terraform.tfvars
echo 'db_password = "Your_DB_Passwd."' >> terraform.tfvars
It’s a good idea to sync terraform modules
terraform get -update
View terraform plan
terraform plan
Apply terraform plan
NOTE: building complete infrastructure may take more than 10 minutes.
terraform apply
Verify instance creation
aws ec2 describe-instances --output table
We are not done yet
Create new AWS CLI profile
In order to use kubectl with EKS we need to set new AWS CLI profile
NOTE: will need to use secret and access keys from terraform.tfvars
cat terraform.tfvarsaws configure --profile terraformexport AWS_PROFILE=terraform
Configure kubectl to allow us to connect to EKS cluster
In terraform configuration we output configuration file for kubectl
terraform output kubeconfig
Add output of “terraform output kubeconfig” to ~/.kube/config-devel
terraform output kubeconfig > ~/.kube/config-develexport KUBECONFIG=$KUBECONFIG:~/.kube/config-devel
Verify kubectl connectivity
kubectl get namespaceskubectl get services
Second part we need to allow EKS to add nodes by running configmap
terraform output config_map_aws_auth > yaml/config_map_aws_auth.yamlkubectl apply -f yaml/config_map_aws_auth.yaml
Now you should be able to see nodes
kubectl get nodes
Working with terraform on EKS
Deploy the Kubernetes Dashboard
Deploy the Kubernetes dashboard
kubectl apply -f \
https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
Deploy heapster to enable container cluster monitoring and performance analysis on your cluster
kubectl apply -f \
https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
Deploy the influxdb backend for heapster to your cluster
kubectl apply -f \
https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
Create the heapster cluster role binding for the dashboard
kubectl apply -f \
https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
Create an eks-admin Service Account and Cluster Role Binding
Apply the service account to your cluster
kubectl apply -f yaml/eks-admin-service-account.yaml
Apply the cluster role binding to your cluster
kubectl apply -f yaml/eks-admin-cluster-role-binding.yaml
Connect to the Dashboard
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}')kubectl proxy
NOTE: Open the link with a web browser to access the dashboard endpoint: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
NOTE: Choose Token and paste output from the previous command into the Token field
Rolling back all changes
Destroy all terraform created infrastructure
terraform destroy -auto-approve
Removing S3 bucket, IAM roles and terraform account
export AWS_PROFILE=defaultaws s3 rm s3://terra-state-bucket --recursiveaws s3api put-bucket-versioning --bucket terra-state-bucket --versioning-configuration Status=Suspendedaws s3api delete-objects --bucket terra-state-bucket --delete \
"$(aws s3api list-object-versions --bucket terra-state-bucket | \
jq '{Objects: [.Versions[] | {Key:.Key, VersionId : .VersionId}], Quiet: false}')"aws s3 rb s3://terra-state-bucket --forceaws iam detach-user-policy --user-name terraform --policy-arn arn:aws:iam::aws:policy/AdministratorAccessaws iam list-access-keys --user-name terraform --query 'AccessKeyMetadata[*].{ID:AccessKeyId}' --output textaws iam delete-access-key --user-name terraform --access-key-id OUT_KEYaws iam delete-user --user-name terraform
Terraform and kubernetes sources can be found in GitHub