Deploying Elasticsearch cluster at AWS ECS – ECS-EC2 terraform module


Hi, AWS fans.

At previous article we examined:

  • terraform modules, related to AWS ECS Elasticsearch deployment, from higher perspective
  • security issues around assume role policies
  • ecs-iam-profile terraform module

At current article we will deploy ec2 instances that would be used as backbone for our ecs cluster. Let’s invistigate ecs ec2 terraform module.

The main logic is located at the file:

resource "aws_instance" "app" {
  ami           = var.image_id
  instance_type = var.instance_type
  key_name      = var.key_name

  vpc_security_group_ids      = []
  availability_zone           =
  associate_public_ip_address = false
  private_ip                  = var.private_ip
  subnet_id = element([
    for s in var.subnets : if s.availability_zone ==
  ], 0)

  disable_api_termination              = false
  instance_initiated_shutdown_behavior = "stop"

  monitoring = false

  iam_instance_profile = var.profile_name

  root_block_device {
    volume_type = "gp3"
    volume_size = var.volume_size
    encrypted   = true

  user_data = data.template_cloudinit_config.ec2.rendered

  tags = merge(local.common_tags, {
    Name = format("%s-%s", local.name_prefix,

We start from  aws_instance resource. The 1st thing is ami (Amazon Machine Image). And here better to choose so called ECS optimized images. You may read more about it here. First of all such an image already has a pre-installed ecs agent, and at second it is already oprimized for ECS. There is a table, where you can find ami id according to your region. Next you can see already familiar things from the bastion article: that is the instance type, ssh key name and all things related to the network, termination behavior. It is completely the same. We also don’t need detailed monitoring for learning purpose. And then there is the line related to the instance profile that we already have created at the previous lecture (iam_instance_profile = var.profile_name). At my udemy course I am paying huge attention to the volumes. Unfortunately it would take me too much time to describe all here. If you are interested at it, then, welcome to my course. At terrafrom code example you can see that EBS would be used as storage for our Elasticsearch data.

But that all was only the tip of the iceberg. A lot of magic happens during instance initialization at users custom data which is rendered from template. Let’s open that template at ecs-ec2/templates/

Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version: 1.0

Content-Type: text/cloud-boothook; charset="us-ascii"

# Set Docker daemon options
cloud-init-per once docker_options sed -i '/^OPTIONS/s/"$/ --storage-opt dm.basesize=25G"/' /etc/sysconfig/docker

Content-Type: text/x-shellscript; charset="us-ascii"

sudo yum update
sudo yum install awscli -y
export AWS_REGION=$(curl -s | grep region | awk -F\" '{print $4}')
export HOST_NAME=$(aws ec2 describe-tags --region=$AWS_REGION --filters "Name=resource-id,Values=$(curl -s" "Name=key,Values=Name" --output=text | cut -f 5)
sudo hostnamectl set-hostname $HOST_NAME

echo 'ECS_CLUSTER=es-cluster' > /etc/ecs/ecs.config
sudo sysctl -w vm.max_map_count=262144
sudo sysctl -w fs.file-max=65536
sudo mkdir -p /usr/share/elasticsearch/data
sudo mkdir -p /usr/share/elasticsearch/config
sudo touch /usr/share/elasticsearch/config/elasticsearch.yml
cd /usr/share/elasticsearch/config

echo \"elasticsearch\" >> elasticsearch.yml
echo node.roles: [ master ] >> elasticsearch.yml
echo $HOST_NAME >> elasticsearch.yml
echo cluster.initial_master_nodes: aws-es-devops-dev-app-a, aws-es-devops-dev-app-b, aws-es-devops-dev-app-c >> elasticsearch.yml
echo bootstrap.memory_lock: true >> elasticsearch.yml
echo >> elasticsearch.yml
echo cloud.node.auto_attributes: true >> elasticsearch.yml
echo discovery.seed_providers: ec2 >> elasticsearch.yml
echo network.publish_host: _ec2_>> elasticsearch.yml
echo transport.publish_host: _ec2_ >> elasticsearch.yml
echo transport.port: 9300 >> elasticsearch.yml
echo http.port: 9200 >> elasticsearch.yml
echo discovery.ec2.endpoint: >> elasticsearch.yml
echo discovery.ec2.availability_zones: eu-central-1a,eu-central-1b,eu-central-1c >> elasticsearch.yml
echo cluster.routing.allocation.awareness.attributes: aws_availability_zone >> elasticsearch.yml
echo discovery.ec2.tag.Elastic: \"node\" >> elasticsearch.yml
echo s3.client.default.endpoint: >> elasticsearch.yml

sudo chown -R 1000.1000 /usr/share/elasticsearch/data/
sudo chown -R 1000.1000 /usr/share/elasticsearch/config/


The syntax provided above is called cloud-init package stuff. Information about cloud init – you can find here at official aws documentation. There is also rather short and clear instruction here at aws blog.

In short, cloud init is a mechanism that allows us to apply any custom logic before an instance would be run. And there is one thing that we have to do. We will often need to expand the root device size volume for production – here you already see how to do it – using root_block_device. Unfortunately, docker daemon will not see any extra space out of the box and it means that our ES instance will not see it properly also. That is why we need to customize it. And we can do it only before the instance is launched. In fact that is the only way we can apply changes we need. Here I used a hardcoded value at 25GB, as I plan to use volumes with 30GB. It is for sure possible to calculate it dynamically – but I decided not to spend my time on that.

Now let’s go further. At every instance we install aws cli which is used to get hostname. As you already noticed, getting such information is very very tricky. The IP address is a “magic” IP in the cloud world, in AWS it is used to retrieve user data and instance metadata specific to an instance. It can only be accessed locally from instances and available without encryption and authentication. At first we are getting an AWS region. Next using ec2 api we get different ec2 instance metadata, where we cut the tag “name”. We define that tag at terraform code above – and that we want to set as hostname. If you are interested in details how all that bash magic works – then, welcome to my course, where I am explaining all it at console step by step.

That is how it appears we do not get the random AWS indentificator for instance’s hostanme – but our custom hostnames e.g aws-es-devops-dev-app-a. Moreover, we also will use hosname value for our node name. And that is how we build our elasticsearch configuration file on the fly at user data from the terraform side.

There are several more essential details here. That is the place where we define the vm_max parameter, which we were already speaking about at the beginning of the current tutorial. Another essential thing which I mentioned is that “While deploying EC2 we have to declare a special custom data for every instance – that will allow ECS agent to register every EC2 instance to ECS cluster”. Here is that special custom information – ECS_CLUSTER=es-cluster – in fact we have to declare only one parameter – ECS_CLUSTER, which should be equal to the ecs cluster name. Now, when we have all that magic behind us, let’s open implementation:

terraform {
  backend "s3" {
    bucket         = "terraform-state-aws-es-devops"
    dynamodb_table = "terraform-state-aws-es-devops"
    encrypt        = true
    key            = "dev-ec2-ecs.tfstate"
    region         = "eu-central-1"

data "terraform_remote_state" "network" {
  backend = "s3"

  config = {
    bucket = "terraform-state-aws-es-devops"
    key    = "dev-network.tfstate"
    region = var.region

data "terraform_remote_state" "iam" {
  backend = "s3"

  config = {
    bucket = "terraform-state-aws-es-devops"
    key    = "dev-iam-profile-ecs.tfstate"
    region = var.region

provider "aws" {
  allowed_account_ids = [var.account_id]
  region              = var.region

module "ecs-ec2" {
  for_each = {
    app-a = {
      private_ip           = "",
      az                   = "eu-central-1a",
      instance_type        = "t3a.small",
    app-b = {
      private_ip           = "",
      az                   = "eu-central-1b",
      instance_type        = "t3a.small"
    app-c = {
      private_ip           = "",
      az                   = "eu-central-1c",
      instance_type        = "t3a.small"

  source = "../../modules/ecs-ec2"

  account_id = var.account_id
  env        = var.env
  project    = var.project
  region     = var.region

  private_ip           = each.value.private_ip
  volume_size          = 30
  key_name             = "dev-ec2-2"
  instance_type        = each.value.instance_type
  az                   =
  image_id             = "ami-06d365907eec04148"
  name                 = each.key

  profile_name = data.terraform_remote_state.iam.outputs.profile_name
  sg      =
  subnets =

As always we define our remote state. We will use IAM remote state for getting ec2 instance profile and network remote state for getting security groups and subnets info. Then we have the list of instances. For every instance we define its type, availability zone and private ip address. Then we loop over the current list and use our ecs-ec2 terraform module for every item.

 Ok, great. Now we are ready to configure our ecs cluster that will use our ec2 instances as an infrastructure for deploying docker containers with Elasticsearch. See you in the next article.

Of course, there much more different terrafrom files, but they are mostly supplementary. To make current article clean and readable I am not going to provide it here. In case you want to download all terraform modules with all files in easy and simple way – welcome to my course at udemy :). As the reader of that blog you are also getting possibility to use coupon for the best possible low price.

architecture AWS cluster devops devops-basics docker elasticsearch flask geo high availability java php programming languages python recommendation systems search systems spring boot symfony