非json格式(低效)的app日志监控 见 grok_plain_text 分支
main分支只维护json格式的app日志
- MacOS 15.6
- Rancher Desktop 1.19
- 本地 K8s 1.33 集群 (2 worker nodes)
- 本地 Harbor 服务器 (harbor.local:80/library/order-service:latest)
- Elastic Stack 8.17
- 收集 非 json 格式的app日志,通过 Filebeat 收集,Logstash清洗, 并存储到 ES
- 使用k8s代替docker compose
- 优化manifest组织结构和使用独立logging命名空间
- json 格式的 app日志 (with logstash-logback-encoder)
- 以 sidecar 模式收集app日志 (with filebeat ndjson parser)
- 收集worker node的日志
- 收集中间件的日志
- 使用kafka做日志缓冲
mvn clean package docker compose up --build
k8s的yaml manifest可以使用 kompose 辅助生成.
- 针对 Filebeat, 需要删除生成的 deployment + pvc, 修改为
daemonset, 让 Filebeat 在每个 Node 上运行一个 Pod,读取/var/log/containers/*.log(K8s 标准日志路径),并自动注入 Pod 元数据(namespace、pod name、labels)。 - 给Filebeat准备service account, 让它可以读取Node 资源, 比如nodeName (add_kubernetes_metadata需要用到)
$ mvn clean package $ docker build -t order-service . $ push order-service $ k get nodes NAME STATUS ROLES AGE VERSION kubeadm-master Ready control-plane 74d v1.33.4 kubeadm-worker01 Ready <none> 66d v1.33.4 kubeadm-worker02 Ready <none> 66d v1.33.4 kubectl apply -f k8s-manifest/logging -n logging kubectl apply -f k8s-manifest/apps -n apps $ k get po -o wide -n logging NAME READY STATUS RESTARTS AGE IP NODE elasticsearch-5599b96998-59fj4 1/1 Running 0 18s 10.244.1.220 kubeadm-worker01 filebeat-2dmfz 1/1 Running 0 18s 10.244.1.217 kubeadm-worker01 filebeat-hjdf4 1/1 Running 0 18s 10.244.2.164 kubeadm-worker02 kibana-d9c96cd58-xd75t 1/1 Running 0 18s 10.244.2.165 kubeadm-worker02 logstash-757b57ff9b-dlrd4 1/1 Running 0 18s 10.244.1.218 kubeadm-worker01 $ k get po -o wide -n apps NAME READY STATUS RESTARTS AGE IP NODE order-service-59678cc76-629w5 1/1 Running 0 24s 10.244.2.166 kubeadm-worker02
# 端口转发 kubectl port-forward svc/order-service 8081:8080 -n apps kubectl port-forward svc/kibana 5601:5601 -n logging # 造http请求 $ siege -c1 -d5 -t60M http://localhost:8081/order/2 # 调式 filebeat 和 logstash的配置文件 (修改cm后,删除pod才能生效) # 不可以使用kubectl rollout restart, 因为cm的修改不会触发pod重启 $ kubectl apply -f k8s-manifest/logging $ kubectl delete po -l k8s-pod=filebeat -n logging $ kubectl delete po -l k8s-app=logstash -n logging
curl -s http://localhost:80/order/2 > /dev/null && tail -n 1 /var/log/app/app.log | jq .
{
"@timestamp": "2025年09月18日T15:16:21.047+0800",
"appname": "demo-elk",
"logger": "a.s.e.OrderController",
"level": "INFO",
"thread": "http-nio-80-exec-1",
"message": "正在处理订单, orderNumber=ORD1758179781047",
"traceId": "57e3d72813f08837ca80ed152227f7bd",
"spanId": "74a75cccd86a0be7"
}注释掉 filebeat-cm-sidecar.yaml 中的 output.logstash, 修改为 output.console, 可以看到被filebeat处理过的spring boot日志输出到控制台
k logs -n apps order-service-57bfd65dc-pk8sq -c sidecar
{
"@timestamp": "2025年09月18日T11:53:11.175Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.17.4"
},
"host": {
"name": "order-service-57bfd65dc-pk8sq"
},
...
}{ "_index": "app-logs-2025年09月18日", "_id": "9UzqXJkBA6sL6YX_QSxn", "_version": 1, "_source": { "@version": "1", "appname": "demo-elk", "thread": "http-nio-80-exec-2", "logger": "a.s.e.OrderController", "traceId": "9e33ddd0a226348c7000b77b363429cb", "spanId": "88aeb3ccce9e7fd8", "level": "INFO", "tags": [ "beats_input_codec_plain_applied", "parsed-success" ], "@timestamp": "2025-09-18T13:01:07.875Z", "message": "正在处理订单, orderNumber=ORD1758200467875" }, "fields": { // OMITTED } }