Terraform OpenSearch module, domain endpoint and dashboard

Hi, devops fans. At 3d part we deal with storage, alarms and variables and finished all preparations. Take my congratulations. Finally you are ready to deploy OpenSearch cluster. First of all you need to apply a network and bastion modules, please do it if you did not perform it earlier. Now let’s go to the opensearch folder and run terraform apply – here what you will see.

You also will see terrafrom changes and question if you want to perform according actions:

Let’s enter yes. Please, be patient – the opensearch cluster deployment can take from 15 to even 25 min – I will pause a video here and return back when it would be finished. As you see in the current execution session it took 18 min to apply terraform. Hard to say why it takes so much time, but for sure it is very annoying :(.

Ok, now let’s see how it looks on the aws console. Let’s find an OpenSearch service. Here you will see the dashboard with the opensearch-dev domain.

Let’s click here and go inside where we can see our cluster configuration. Here is the vpc endpoind which is the address which we can use to communicate with our cluster inside our network.

Let me show what I mean. Let’s deploy our bastion and enter at it using ssh. Now we can perform requests to the opensearch cluster using programming languages and REST API, or we can use a simple curl package.

Small notice for people who want to make integration with opensearch cluster using some programming language. In most cases you will use some framework and ready ES library that will require passing a so-called connectioin string. Domain endpoint – that is also your connection elasticsearch string. But remember to add 443 port as by default, in most cases, libraries are using 9200 port. E.g in the current case we have to use {endpoint}:443. If you are interested at how to integrate ES using such popular programming solutions as Java And Spring Boot, or PHP and Symfony, or Python + Flask – then I engage to visit my course “Elasticsearch as you have never known it before” where I am describing it in details

Ok, now I want to show you one more interesting tool – that is the OpenSearch dashboard. That provides UI for working with cluster from the browser. If you will simply click at the current url – you will find that it is not working.

The problem here is the permissions. It will be working in case you will put your cluster on a public network – but it is not always the case we want to have. In most cases you will probably want to hide the opensearch cluster on a private network, as we have it in the current case. You may expose a dashboard exposing some proxy e.g nginx, which is a recommended solution according aws documentation, but I am going to show one trick how you can do it faster and easier. The simple workaround for it is using the ssh tunnel. Here is how we can define it using our bastion and ssh config:

Host estunnel
    HostName x.x.x.x  #chenage it at bastion IP address
    User ubuntu
    IdentitiesOnly yes
    IdentityFile ~/.sergii-blog-keys/dev-bastion-2
    LocalForward 19999 {put_your_vpc_endpoint_here}:443

So let’s input localhost:19999/_dashboards/ and you have to see the dashboard by itself. You will see here ssl certificate warning – but it is not essential as traffic is not goint to the internet – it is mapped directly to our local machine. So we can ignore it.

Lets click at “Dev tools” menu as it is shown at screen above. You will see next view:

So in such a way you can perform any operations at your OpenSearch cluster. What is the most interesting – that OpenSearch dashboard – that is in fact Kibana alternative – the same as OpenSearch by itself is an Elasticsearch alternative. It is not an article about data visualization, so I am not going to talk here a lot about it. I am showing a little bit more about it at my udemy course. But as a short summary I can say that in case visualization you are limited here only buy your own imaginations. If you are interested more about the OpenSearch panel I recommend starting from AWS documentation. There is also a nice tutorial. And opensearch dashboard documentation by itself. That is all I want to tell you at current article. If you want to get more about OpenSearch or Elasticsearch, then welcome to my online course:

ENROLL AT AWS devops course