Hi, devops fans. Here I am opening the series of articles devoted to the theme of how to deploy high available Elasticsearch cluster using AWS service, which is called OpenSearch, and terraform. According to AWS official documentation current service makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. We will check it in practice and deploy an OpenSearch HA cluster using terraform. We will also discuss if OpenSearch is indeed the Elasticsearch alternative. I will share with you some interesting facts that appeared from my commercial experience of using AWS OpenSearch. I also will explain different details of the OpenSearch terraform module which we will create together. After deploying the OpenSearch cluster we will make an overview of different management and monitoring OpenSearch AWS tools.
But before starting with realizing deployment scenario we have to make some initial preparations. We will start our work from the “Initial terraform configuration aspects”, where we will prepare and set environment. After that we will run our 1st pre-init terraform module, which will create s3 bucket for keeping terraform state, and DynamoDB table for keeping terraform locks within the best DevOps practices.
After that we will create some base AWS skeleton configuration. And again we will use terraform modules for that – only infrastructure as a code. We will examine and apply:
1. Network module, that will create vpc, subnets, route tables, network address translation (NAT), internet gateways (IG) and prepare security groups (SG). I also want to make an accent here – current skeleton is made for the purpose of the learning. It will allow to pass over different deployment scenarios of Elasticsearch at AWS. You may use it in production – but it will require some adjustments – that is not a silver bullet for everything in the world. For example, NAT gateway is not required for OpenSearch. It is quite possible that you may need only a private network at all – that all depends on your requirements. But the current skeleton is a good point where you can start from. In most cases you will need to make only some small adjustments in order to use it at production, e.g clean code from excessive stuff or modify security groups. Our realization architecture scheme will looks like as:
2. Then we will go further with bastion module – which will create bastion host that would allow us to connect to our infrastructure in a safe way according to best security practices. You may ask- why not SSM (Sytems Manager Agent) ? It is reasonable question. In the case of Elasticsearch it is very useful to make different ssh forward tunnels, e.g in order to connect cluster to Elasticsearch plugin or open OpenSearch dashboard for local usage. Personally for me, it is much more convenient to do it using ssh client and ssh config. It is also not a problem to manage bastion via terraform. Though if you prefer SSM, then you may perform the same actions using it.
Below is the list of different articles related to the the topic “How to deploy HA Elasticsearch cluster at AWS using OpenSearch”:
- Two main questions: what is OpenSearch, if OpenSearch and Elasticsearch are compatible?
- Initial terraform environment and pre-init terraform module
- Terraform network module
- Terraform bastion module
- Terraform OpenSearch module
New articles would be added gradually, so if you are interested at current topic, please visit that page regularly or subscribe to my newsletter. But if you are not ready to wait – then I propose you to view all that material at my on-line course at udemy. Below is the link to the course. As the reader of that blog you are also getting possibility to use coupon for the best possible low price.
