AWS devops: ElasticSearch at AWS with terraform and ansible


Are you interested in docker orchestration? Would you like to uncover deep insights and pitfalls of AWS? Do you want to see the real power of Terraform and Ansible at real commercial examples? Or maybe you are looking for materilas that will help you to arrange your devops theory knowledge via the prism of practice. If yes –  then you are definitely at the right place, as here I want to represent my course where you can study all aboved mentioned areas while learning how to deploy HA Elasticsearch cluster at AWS using different scenarios and devops tools.   

Hi, my name is Sergii Demianchuk. I have over 17th year experience in different IT areas. Currently I am working as CTO and Architect at one of the biggest Europe transport marketplaces – Clicktrans. Here I want to represent my 2d course devoted to ES and different devops aspects around it. The 1st one course (Elasticsearch as you have never known it before), which was devoted to ES basics, how to work with ES using popular programming languages and frameworks, appeared to be very successful – it gathered over half 1000 students within 8 months, got very positive feedback, and the blog around the course gathered over 20K users. But there is one problem: students from my 1st course, the same as readers of my blog, constantly ask me different questions related to devops issues around ES. That is why that course appeared and it is called “AWS devops: ES at AWS using terraform and ansible.“

First of all I want to note that the current course is not for beginners. To start it you should know terraform/ansible basics, you need to have an AWS account and understand basic AWS conceptions, e.g VPC, EC2, security groups… You also have to be good at docker and to be familiar with docker orchestration. It is not the course where you will find coding exercises on how to write TODO terraform or ansible modules. At that course you will get ready working solutions, used in commercial life that can be easily extended to new requirements.

The course consists of 5 main modules. 

The first one is about how to deploy ES as an AWS service, which is called OpenSearch. Here I will tell you about interesting OpenSearch pitfalls that I discovered during my commercial practice. And believe me – you  will not find that information at AWS documentation. You will get a ready terraform module for deploying the Opensearch cluster and we will analyze it step by step together.

In the 2d part I will show how to deploy an ES cluster at AWS ECS (Elastic Container Service). And again we will use the terraform modules for that. I will show you why you can’t use Fargate in that case and how many problems should be resolved to deploy ES at ECS.

Next course section is about how to deploy ES at AWS EC2 instances using docker swarm. I will prove to you that it is the easiest and at the same time the most cheap way to deploy ES at almost any cloud for most small and middle companies with a steady growth pattern. You will get ready terraform modules that will allow you to prepare AWS infrastructure and ansible modules for deploying HA ES cluster upon it. 

The 4th part is devoted to autoscaling possibilities of ES cluster using different deployment approaches. We will talk about what we really can autoscale, when we can use it and what big limitations we have in case of working with ES.

5th module is devoted to the theory. If you are already familiar with ES cluster conceptions and everything that is related to  shards and ES performance – then you can skip it. But if not –  that is the place where you have to start from. As the reader of that blog you are also getting possibility to use coupon for the best possible low price.

About the author


Software engineer with over 15 year’s experience. Everyday stack: PHP, Python, Java, Javascript, Symfony, Flask, Spring, Vue, Docker, AWS Cloud, Machine Learning, Ansible, Terraform, Jenkins, MariaDB, MySQL, Mongo, Redis, ElasticSeach

architecture AWS cluster devops devops-basics docker elasticsearch flask geo high availability java php programming languages python recommendation systems search systems spring boot symfony