Thanks for the PR

it's the recommended way to setup the Elasticsearch Java client!

I wasn't aware of that. How does it work with Testcontainers and Docker compose then? Both of them retrieve a password from the environment. I can see you have updated ElasticsearchConnectionDetails but none of their two implementations.

Creating the API key has to be done through either an API call or in the dedicated Kibana menu, this is because you can decide to create one with certain privileges, or with an expiration date. So the default for both implementations of ElasticsearchConnectionDetails would be null, because the API key cannot be retreived from the environment, but only created after the elasticsearch instance is running.

But it works the exact same for cloud, testcontainers or docker, it's just an additional step unlike username and password credentials.

@l-trotta and @snicoll maybe we can simplify the Elasticsearch testing using start-local. This script installs Elasticsearch with a .env file containing all the information for the connection, including the API key (ES_LOCAL_API_KEY). If you only need Elasticsearch and not Kibana you can use the --esonly option.

Hi, what about providing and env var in Elasticsearch image which allows to configure the API Key? That would benefit Docker Compose and Testcontainers implementations. Also, if this is the recommended way then the image could provide a default way to enable this.

On the Testcontainers side we can add a custom implementation to generate the API Key but I am more in favor to provide the same experience for Docker Compose and Testcontainers users by just enabling the API key with a env var.

@eddumelendez Looking at what @l-trotta pointed out, it seems that one of the main driver behind API keys is to have fine-grained roles per index, since you need to provide those during API key generation. I guess that for a given user, you want to use different keys for specific usage patterns and restrict roles accordingly.
Is this something that testcontainers is already supporting in some way, or is it something that developers should write as part of their tests?

Hi @bclozel, currently, developers must extend and override ElasticsearchContainer, implement containerIsStarted as part of the Testcontainers Lifecycle and make a similar call to start-local using a HTTP library.

I've uploaded an example using Testcontainers and Elasticsearch using API Key.

I've placed this on hold for now as I'd like to tackle the migration to Rest5Client before we add any more configuration options.

@wilkinsona do you have any update on the migration to Rest5Client? The PR seems to be merged. Thanks.

Yes, that work's complete now. Next step would be to revisit the proposal here on top of that new base.

Thanks @wilkinsona

@ezimuel I don't think you mean to close this PR, isn't it? That happens when you create it from main and you force push which would close the PR. It's shame as we'd have liked to keep the conversation history...

@snicoll I'm working on it! will push the new code ASAP and reopen it

Thanks for the updates, @l-trotta. What do you think should happen if the user configures both an API key and a username and password? Should we:

• Configure the client with only the API key?
• Configure the client with only the username and password?
• Configure the client with both?
• Fail?
• Something else?

Hey @wilkinsona! Currently the java client builder helper (ElasticsearchClient.of()) fails with java.lang.IllegalArgumentException: Cannot set both username/password and API key, but there are no checks for the extended builder, which Spring uses. Should we apply the same logic here so that it's coherent? Sending both Basic and ApiKey header to the server won't result in an error, the first valid one will be used.

Thanks, @l-trotta.

Once a property has been set in Spring Boot, it can be a bit harder than we'd like to then unset it (#24133). If we made the API key and the username and password mutually exclusive this could make it hard for a user to override one with the other. For this reason, I think we should leave things as they are and allow both to be configured. We can always refine this in the future if it proves to be problematic.

good to know @wilkinsona! everything's ready then

Thanks very much, @l-trotta.