Elasticsearch as a service - AWS vs Azure vs Google

Elasticsearch is a powerful search and analytics engine that can be used to index, search, and analyze large volumes of data in real-time. Elasticsearch can be deployed on-premises, but many cloud platforms now offer Elasticsearch as a service. In this article, we’ll compare different well-known cloud platforms that provide Elasticsearch as a service, based on their pricing, size, and ease of deployment from a technical perspective. The date of comparison is 07.04.2023. After Elasticsearch project has changed the licensing terms for it’s software, which happened at in the middle of 2021, world wide Cloud providers had to deal with that somehow. AWS made a fork form opensource Elasticsearch repository and started it’s own software product based on it. While Google and Azure refused from fully managed Elasticsearch service and chose the road of partnership. That articles tries to compare above mentioned cloud provider to bring some light at current situation around Elasticsearch and using it as a service.

Amazon OpenSearch Service

Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. Amazon OpenSearch Service supports OpenSearch and legacy Elasticsearch OSS (up to 7.10, the final open source version of the software). When you create a cluster, you have the option of which search engine to use. Amazon OpenSearch Service offers multiple instance types, including T2-T3, M4-M6, R3-R6, C4-C6, and provides automatic scaling to meet the demands of your applications. In case auto scaling you have different options, including blue/green deployment that allows you replace existing cluster with completely new one without any pauses. Amazon OpenSearch Service also offers a range of security features, such as encryption at rest and in transit, and integrates with other AWS services like AWS Identity and Access Management (IAM).

Pricing for Amazon Elasticsearch Service is based on the number of instances you use and the amount of data you store. The pricing starts at $0.018 per hour for T2 micro instances (1 vCPU, 1 GB RAM) and goes up from there. The better way to estimate costs is to use AWS pricing calculator

From a technical perspective, deploying Elasticsearch on Amazon OpenSearch Service is straightforward, but there are a lot of details and pitfalls. You can create an Elasticsearch domain using the AWS Management Console, the AWS CLI, or the Amazon OpenSearch Service API. Amazon OpenSearch Service provides built-in OpenSearch dashboards (Kibana “brother”) for data visualization and access control. You also can easily integrate Elasticsearch with other AWS services like Amazon S3 and Amazon CloudWatch. If you are interested more about Elasticsearch, OpenSearch and AWS – here are some more detailed articles from my blog related to it:

Google Cloud Elasticsearch

There is no fully managed Google Cloud Elasticsearch standalone service. Instead of it Google propose integration with Elastic Cloud via it’s marketplace. There are a lot of integration solutions that allow to inject data from GCP Data platforms like Pub/Sub, BigQuery and CloudStorage. From technical perspective, deploying Elasticsearch is not complicated, though number of possible subscriptions and configuration options is tremendous and it requires a lot of time to deal with all that options. Pricing policy, as for me, is completely non intuitive and over complicated as virtual Elastic Billing units are used.

The best way to estimate costs, from my perspective, is ti use Elastic Cloud calculator. In case using standard subscription the Elasticsearch node with 1-2 vCPU, 1 GB RAM parameters will cost $0.0274 per hour

Microsoft Azure Elasticsearch

In case Azure Microsoft Cloud, situation is completely the same – Azure provide integration with Elastic Cloud in the same way as Google has it. And again better to use elastic cloud calculator to estimate costs.

In case using standard subscription the Elasticsearch node with 1-2 vCPU, 1 GB RAM parameters will cost $0.0303 per hour

Now let’s assume that we want to built some search system that would be used by some popular e-commerce web site. After tests it appeared that we need have 3 node HA Elasticsearch cluster. To minimize costs we want to use data nodes that are performing master’s role at the same time. We also know that required resources for stable work per node is 8GB RAM, 2 CPU, plus we need ~200 Gb of storage. Lets compare cost at that case. So, in case AWS we get $346.00 per month

Now Google + Elastic Cloud integration – we get: 0.6576 (hour rate) x 24 hours x 30 days per month = $473,472 per month. Please, pay attention that I am choosing hot data option, which is much more cheaper then Enterprise Search, that would be more preferable in current test scenario.

In case Azure Microsoft + Elastic Cloud integration we get: 0.7272 (hour rate) x 24 hours x 30 days per month = $523,584 per month

Let’s make some small summary table

Cloud Platform	Pricing for 3 node cluster	Scalability	Ease of Deployment
AWS OpensSearch service	$346.00 per month	Automatic scaling for data nodes, blue/green deployment also available	Straightforward
Google Cloud + Elastic Cloud	$473.00 per month	Automatic scaling for data nodes	Straightforward
Azure Microsoft + Elastic Cloud	$523.00$ per month	Automatic scaling for data nodes	Straightforward

So, as you can see, AWS looks like much more attractive from price perspective then Google or Azure Cloud. In case AWS, as for me, we also have much more flexibility in management and auto scaling. For sure, which cloud platform to choose depends on your specific needs. Especially it depends on whether you are already strongly engaged at using some exact cloud platform or not. But, from my point of view, partnership layer brings additional layer of complication, it also raises cost of service in the end. Anyway, cost of Elasticsearch software as a service is still very high even for bigger companies. Looking at prices, no wonder, that most of companies still prefer to support self managed Elasticsearch cluster either at premises or at virtual cloud machines.

P.S.
If you are interested at how to deploy HA Elasticsearch cluster at AWS using different scenarios and devops tools, then welcome to my online course: “AWS DevOps: Elasticsearch at AWS with terraform and ansible“

Best regards