Connect APM server to elasticsearch

Question

I am trying to send telemetry data from ASP.NET app to elastic search. (logs, metrics, traces).

I tried to log data without APM. Logs are sent to elastic without APM, but trace does not work. To support traces I found that APM is necessary. Right now, I am trying to send all data to otel-collector which send data to APM server and APM server should send data to Elasticsearch instance.

Unfortunately, APM server could not send data to Elastic search and logs following:

{"log.level":"error","@timestamp":"2025年09月30日T12:47:08.014Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error querying cluster_uuid: status_code=401","service.name":"apm-server","ecs.version":"1.6.0"}

What should be fixed in this configuration to make it work ?

• If there is alternative, which excludes APM server, I would prefer that method. Including one more item in my infrastructure does not make me happy.

Infrastructure is managed by docker-compose. here is my docker compose file

services:
 elasticsearch:
 image: elasticsearch:9.1.4
 container_name: elasticsearch
 environment:
 - discovery.type=single-node
 - xpack.security.enabled=true
 - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
 - ELASTIC_PASSWORD=Password123
 ports:
 - "9200:9200"
 volumes:
 - es_data:/usr/share/elasticsearch/data
 networks:
 - elastic
 kibana:
 image: kibana:9.1.1
 container_name: kibana
 environment:
 - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
 - ELASTICSEARCH_SERVICEACCOUNTTOKEN=AAEAAWVsYXN0aWMva2liYW5hL2tpYmFuYV9zZXJ2aWNlX3Rva2VuOmtkb3F2NnZxVEFHQXFrcEZDVDdYNlE
 - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=46d597016b5bdda4374240b48d07d39a
 ports:
 - "5601:5601"
 depends_on:
 - elasticsearch
 networks:
 - elastic
 # 🚀 NEW SERVICE: Elastic APM Server
 apm-server:
 image: docker.elastic.co/apm/apm-server:9.0.0 # Use the same major version as Elasticsearch/Kibana
 container_name: apm-server
 environment:
 # Output to Elasticsearch
 - output.elasticsearch.hosts=["elasticsearch:9200"]
 - output.elasticsearch.username=elastic
 - output.elasticsearch.password=Password123
 # Enable OTLP intake (OpenTelemetry Collector sends data here)
 - apm-server.host=0.0.0.0:8200
 - apm-server.sampling.keep_uncategorized_services=true
 ports:
 # OTLP endpoint for the collector (OpenTelemetry)
 - "8200:8200" 
 depends_on:
 - elasticsearch
 networks:
 - elastic
 # The OpenTelemetry Collector now depends on APM Server
 otel-collector:
 image: otel/opentelemetry-collector-contrib:latest
 container_name: otel-collector
 command: [--config=/etc/otelcol/config.yaml]
 volumes:
 - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
 ports:
 - "4317:4317" # OTLP gRPC
 - "4318:4318" # OTLP HTTP
 # ADD DEPENDENCY ON APM SERVER
 depends_on:
 - elasticsearch
 - apm-server
 networks:
 - elastic
volumes:
 es_data:
 driver: local
networks:
 elastic:
 driver: bridge

here is my otel-collector config

receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318
exporters:
 otlp/elastic_apm:
 endpoint: "http://apm-server:8200"
 tls:
 insecure: true
 insecure_skip_verify: true
processors:
 batch:
 # Add these processors for better trace handling
 memory_limiter:
 limit_mib: 512
 check_interval: 1s
 resource:
 attributes:
 - key: service.instance.id
 value: ${HOSTNAME}
 action: upsert
service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [resource, batch]
 exporters: [otlp/elastic_apm] 
 metrics:
 receivers: [otlp]
 processors: [batch]
 exporters: [otlp/elastic_apm]
 logs:
 receivers: [otlp]
 processors: [batch]
 exporters: [otlp/elastic_apm]