Splunk otel collector unable to process file log receiver

Question

I am trying to send Azure Devops agent logs to cluster receiver.

Here is the receiver config block

apiVersion: v1 
 kind: ConfigMap 
 metadata: 
 name: otel-sidecar-config 
 namespace: xxxxxxxxxx 
 data: 
 config.yaml: | 
 receivers: 
 filelog: 
 include: 
 - /azp/\**.log 
 start_at: beginning 
 poll_interval: 200ms 
 retry_on_failure: 
 enabled: true 
 include_file_path: true 
 include_file_name: true 
 operators: 
 - type: recombine 
 id: azdo-recombine 
 source_identifier: attributes\["log.file.path"\] 
 combine_field: body 
 combine_with: "\\n" 
 is_first_entry: 'body startsWith "\["' 
 max_log_size: 1048576 
 output: parser-azdo 
 - type: regex_parser 
 id: parser-azdo 
 regex: '^\[(?P\[0-9\]{4}-\[0-9\]{2}-\[0-9\]{2} \[0-9\]{2}:\[0-9\]{2}:\[0-9\]{2}Z)\\s+(?PINFO|WARN|VERB|ERROR)\\s+(?P\[^\\\]\]+)\]\\s+(?P\[\\s\\S\]*)$' 
 timestamp: 
 parse_from: attributes.timestamp 
 layout_type: gotime 
 layout: '2006-01-02 15:04:05Z' 
 on_error: send 
 output: azdo-severity 
- type: severity_parser 
 id: azdo-severity 
 parse_from: attributes.severity 
 mapping: 
 trace: \[VERB\] 
 info: \[INFO\] 
 warn: \[WARN\] 
 error: \[ERROR\] 
 output: finalize 
 - type: move 
 id: finalize 
 from: attributes.message 
 to: body 
 processors: 
 memory_limiter: 
 check_interval: 1s 
 limit_mib: 400 
 spike_limit_mib: 100 
 k8sattributes: 
 extract: 
 metadata: 
 - k8s.pod.name 
 - k8s.namespace.name 
 resource: 
 attributes: 
 - key: service.name 
 from_attribute: k8s.pod.name 
 action: upsert 
 batch: {}
 exporters:
 otlp:
 endpoint: otel-cluster-receiver.splunk.svc.cluster.local:4317
 tls:
 insecure: true
 debug:
 verbosity: detailed
 
 service:
 telemetry:
 logs:
 level: debug
 pipelines:
 logs:
 receivers: [filelog]
 processors: [memory_limiter, batch]
 exporters: [otlp]

Here is the errors information:

2025年12月15日T17:37:35.848Z error helper/transformer.go:154 Failed to process entry {"resource": {"service.instance.id": "5778d025-1300-4d5e-bd75-bf3645a20f8e", "service.name": "otelcol", "service.version": "v0.140.0"}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "operator_id": "parser-azdo", "operator_type": "regex_parser", "error": "regex pattern does not match", "action": "send", "entry.timestamp": "0001年01月01日T00:00:00.000Z", "log.file.name": "Agent_20251215-173735-utc.log", "log.file.path": "/azp//Agent\_.log"} 
GitHub
2025年12月15日T17:37:35.848Z error helper/transformer.go:154 Failed to process entry {"resource": {"service.instance.id": "5778d025-1300-4d5", "service.name": "otelcol", "service.version": "v0.140.0"}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "operator_id": "azdo-severity", "operator_type": "severity_parser", "error": "log entry does not have the expected parse_from field: {"parse_from":"attributes.severity"}", "action": "send", "entry.timestamp": "0001年01月01日T00:00:00.000Z", "log.file.name": "Agent_2025121.log", "log.file.path": "/azp/Agent_20121.log"} 
GitHub
2025年12月15日T17:37:35.848Z error helper/transformer.go:154 Failed to process entry {"resource": {"service.instance.id": "5778d025-1300-4d5e-bd75", "service.name": "otelcol", "service.version": "v0.140.0"}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "operator_id": "finalize", "operator_type": "move", "error": "move: field does not exist: attributes.message", "action": "send", "entry.timestamp": "0001年01月01日T00:00:00.000Z", "log.file.name": "Agent_20251215-173735-utc.log", "log.file.path": "/azp//Agent_20251.log"} 
 

The is log format form the file::

[2025年12月15日 11:58:13Z INFO AgentCapabilitiesProvider] Adding 'Agent.OS': 'Linux' 
[2025年12月15日 11:58:13Z INFO AgentCapabilitiesProvider] Adding 'Agent.OSArchitecture': 'X64' 
[2025年12月15日 11:58:13Z INFO AgentCapabilitiesProvider] Adding 'InteractiveSession': 'True' 
[2025年12月15日 11:58:13Z VERB VisualStudioServices] Created OAuth issued token provider instance 65942258 (Bearer) 
[2025年12月15日 11:58:13Z WARN VisualStudioServices] Authentication failed with status code 401. 
Cache-Control: no-cache 
Pragma: no-cache 
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT" 
WWW-Authenticate: Bearer authorization_uri=[https://logi](https://logi/), Basic realm="https:///", 
TFS-Federated 
X-TFS-ProcessId: 4983132d-1814-4d88-9060-cf367f589768 
Strict-Transport-Security: max-age=31536000; includeSubDomains 
ActivityId: 3485d06f-

• Ranadip Dutta: I dont see Regex matching your lines (group names invalid / escaping off). attributes.severity and attributes.message don’t exist, so the severity and move operators fail. Since you didn’t parse it into the entry’s timestamp, I believe it will consider the default.
• ramesh reddy: yeah, this regex not working. '^[(?P[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}Z)\\s+(?PINFO|WARN|VERB|ERROR)\\s+(?P[^\\]]+)]\\s+(?P[\\s\\S]*)$' any suggestions on this ?

Answer 1

I have updated the regex. Hopefully, it will do the work:

^\[(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}Z)\s+(?P<severity>INFO|WARN|VERB|ERROR)\s+(?P<component>[^\]]+)\]\s+(?P<message>[\s\S]*)$
``

There were many unnecessary escapes and `attributes.timestamp` will work now because the named group exists now.

For severity, you should use plain token instead of brackets. This is the regex validated output:

Output:

timestamp=2025年12月15日 11:58:13Z
severity=INFO
component=AgentCapabilitiesProvider
message=Adding 'Agent.OS': 'Linux' 

Updated Config file:

apiVersion: v1
kind: ConfigMap
metadata:
 name: otel-sidecar-config
 namespace: xxxxxxxxxx
data:
 config.yaml: |
 receivers:
 filelog:
 include:
 - /azp/**/*.log 
 start_at: beginning
 poll_interval: 200ms
 retry_on_failure:
 enabled: true
 include_file_path: true
 include_file_name: true
 # Stanza operators must be defined under the filelog receiver
 operators:
 
 - type: recombine
 id: azdo-recombine
 source_identifier: attributes["log.file.path"]
 combine_field: body
 combine_with: "\n"
 is_first_entry: 'body startsWith "["'
 max_log_size: 1048576
 output: parser-azdo
 # Parse timestamp, severity, component, and message from the bracketed first line
 - type: regex_parser
 id: parser-azdo
 parse_from: body
 regex: '^\[(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}Z)\s+(?P<severity>INFO|WARN|VERB|ERROR)\s+(?P<component>[^\]]+)\]\s+(?P<message>[\s\S]*)$'
 timestamp:
 parse_from: attributes.timestamp
 layout_type: gotime
 layout: '2006-01-02 15:04:05Z'
 on_error: send
 output: azdo-severity
 # Map severity strings to OTLP severity number/text
 - type: severity_parser
 id: azdo-severity
 parse_from: attributes.severity
 mapping:
 trace: VERB
 info: INFO
 warn: WARN
 error: ERROR
 output: finalize
 # Move parsed message into the log body for downstream exporters
 - type: move
 id: finalize
 from: attributes.message
 to: body
 processors:
 memory_limiter:
 check_interval: 1s
 limit_mib: 400
 spike_limit_mib: 100
 k8sattributes:
 extract:
 metadata:
 - k8s.pod.name
 - k8s.namespace.name
 
 
 # If you prefer the original service.name, remove resource.attributes below.
 resource:
 attributes:
 - key: service.name
 from_attribute: k8s.pod.name
 action: upsert
 batch: {}
 exporters:
 otlp:
 endpoint: otel-cluster-receiver.splunk.svc.cluster.local:4317
 tls:
 insecure: true
 debug:
 verbosity: detailed
 service:
 telemetry:
 logs:
 level: debug
 pipelines:
 logs:
 receivers: [filelog]
 processors: [memory_limiter, k8sattributes, batch]
 exporters

Glob: /azp/**/*.log (clean double-star recursive) instead of /azp/\**.log and Add k8sattributes to the logs pipeline processors and include the debug exporter so you can see parsed entries in the Collector logs.

Hope it helps

• ramesh reddy: i am still getting same error. i used this. but i am surprised to see it got added \\ every where automatically regex: '^\[(?P<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}Z)\\s+(?P<severity>INFO|WARN|VERB|ERROR)\\s+(?P<component>[^\]]+)\]\\s+(?P<message>[\\s\\S]*)$'
• Ranadip Dutta: My suggestion would be to inspect the live ConfigMap kubectl -n xxxxxxxxxx get cm otel-sidecar-config -o yaml | sed -n '1,200p'
• Ranadip Dutta: If still you are getting \\ then use a block scalar for the regex (as above) to avoid quoting/escaping rules entirely. Or, if you must keep double quotes in your tooling, ensure the generated YAML still contains single backslashes in the final ConfigMap (check with kubectl get cm).