Magento 2.4 with AWS ES not able to install

Question

I'm trying to install Magento 2.4. I have set up AWS ES that I tasted and it works fine! But I keep getting an error:

 In SearchConfig.php line 81:
 Could not validate a connection to Elasticsearch. null

Is there a special configuration I need to make on AWS es?

Thank you.

• paj: are you specifying the correct credentials for elastic search in your install command line arguments? See here for more devdocs.magento.com/guides/v2.4/install-gde/install/cli/…
• codexxx: yes the credentials are correct.
• paj: aws access control corretcly configured? Can you manually connect from the M2 server command line with curl?
• codexxx: yes I can manually connect from M2 server using curl

Answer 1

I actually spent the last 2 days working on this and I got it to work. You could open AWS ES to the world and call it a day, but of course that's not safe at all.

What I have done is create a new ES domain with VPC access, no fine–grained access control and allow open access to access policy. Now you want only the servers to have access to the ES domain. Create a new Security Group giving 443 access to the Security Group that the servers are part of. By doing this you limit the access to the WEB servers and you don't need an username and password to connect. To connect to AWS ES with username/password you need to send some headers and I am afraid Magento doesn't have this configured out of the box.

But if you decide to go with the steps above, you just need to add this to the Magento installation:

--elasticsearch-host="https://vpc-YOURAWSES-somerandominformation.us-east-1.es.amazonaws.com" \
--elasticsearch-port=443

I am using port 443 because I enable to only accept HTTPS connections.

Just keep in mind that by doing this you won't have access to Kibana using public internet, as being on the VPC means it's on private network. You could create a VPN gateway and connect to it if you really want Kibana access.

Another option is to create a master user in the fine–grained access control option and use the following on the installation:

--elasticsearch-host="https://vpc-YOURAWSES-somerandominformation.us-east-1.es.amazonaws.com" \
--elasticsearch-port=443 \
--elasticsearch-username magento \
--elasticsearch-password 'Thii$4ドルmePassword!'

As you can see I added single quotes to the password, because AWS ES needs special characters. With this setup you will be able to access Kibana.

I personally want minimum access from the world and don't care to access Kibana for the indexes. I have used Kibana/ES for Magento logs before, but I have found it to be problematic, so I just used EFS for the log folders and called it a day.

Let me know if this works out.

• Eric Cavalcanti: After almost 2 years I had the same issue with AWS OpenSearch and this is still valid. The only thing I couldn't figure out was the fine-grained access control with master user, so I decided to go without password and lock access to the servers I know need access.

Answer 2

i think on aws you can connect to host https://elk-end-point port 443

also you need to use latest AWS ELK 7.6+ stack

Answer 3

Yes, there is a specific technology stack requirements that you must follow, there are PHP extensions and third-party system to install (e.g. ElasticSearch 7, MySQL 8, etc..).

You will be able to find all the Magento 2.4 technology stack requirements in the link.

The issue that you are having is caused by the missing of the communication with ElasticSearch 7.

In order to test your ElasticSeach, you can run this command below, if needed, you can change the localhost to your ElasticSearch IP and port.

curl -XGET 'localhost:9200/my_index/my_type/_search' -d '{
 "query" : {
 "match" : {
 "testField" : "abc"
 }
 }
}'

Answer 4

This is meant to be a comment to Eric Cavalcanti's 2020 answer, but I don't have enough reputation.

It's 2024, and I'm setting up Magento 2.4.7-p1, and AWS is providing OpenSearch 2.13.

I could not get fine-grained access control to work. With the help of participants (Scott, Nate - HUGE THANK YOU) on the Magento Slack community I determined that I had to:

1. Restrict access to "VPC access" Image of Network settings
2. create the domain without fine-grained access control
3. choose "Configure domain level access policy"
4. ensure that I set the access policy to "Allow" Image of Access policy settings

Then, when running bin/magento setup:install these are the search related options I used

--search-engine=opensearch \
--opensearch-host=https://vpc-MyDomain-AwsRandomChars.MyAwsRegion.es.amazonaws.com \
--opensearch-port=443 \