v-agent

Overview

v-agent is a daemon that gathers metrics and writes them to a compatible remote_write endpoint.

It acts like prometheus (scraping metrics), re-labeling metrics, then pushing them off to a remote_write compatible endpoint. Internally, we use mimir and write our metrics there but other implementations should work.

This is a modified version of v-agent that allows it to run anywhere removing the cloud providers "block".

Metrics

All metrics that are specifically created with v-agent are prefixed with v_. Scraped metrics are not modified other than the addition of labels.

Every metric will have all metrics in labels_config added to it. The following are special labels:

• hostname: Pulled automatically. Set with HOSTNAME environment variable or os.Hostname()

The above labels are added to ensure that the metric is unique. A lack of uniqueness can result in metrics getting overwritten/clobbered.

System metrics that are collected:

• CPU utilization: system, user, steal, utilization, etc.
• Memory utilization: cached, buffered, utilization, etc.
• Load average: 1, 5, 15, and tasks
• Disk stats: writes/reads, etc.
• Filesystem stats: bytes, inodes, utilization
• NIC: bytes, packets, errors, etc.

Kubernetes:

• v_kube_apiserver_healthy that is 0 (if healthy) or 1 if not healthy based on response from kube-apiserver /healthz endpoint.
• Every metric from /metrics

Etcd:

• v_etcd_healthy that is 0 (if healthy) or 1 if not healthy based on response from etcd /health endpoint.
• Every metric from /metrics

Konnectivty:

• v_konnectivity_healthy that is 0 (if healthy) or 1 if not healthy based on response from konnectivity /healthz endpoint.
• Every metric from /metrics

HAProxy:

• v_haproxy_healthy that is 0 (if healthy) or 1 if not healthy based on response from /metrics endpoint.
• Every metric from /metrics

Ceph:

• v_ceph_healthy: Not implemented yet.
• Every metric from /metrics

Usage

Configuration is through config.yaml, sample:

debug:true# debug outputinterval:60# interval to scrape metricsendpoint:https://endpoint... # remote endpointbasic_auth_user:""# basic auth userbasic_auth_pass:""# basic auth passcheck_vendor:false# when true, vendor must be "Vultr"; set to false otherwiselabels_config:# any labels below will be added to all metricshostname:""# empty string uses local hostname, unset (nil) doesnt use, non-empty string uses specified labelany:any # any key/value labelprobes_api:listen:0.0.0.0port:7091metrics_config:agent:load_avg:enabled:truecpu:enabled:truememory:enabled:truenic:enabled:truedisk_stats:enabled:truefilter:"sr0"# regexfile_system:enabled:truekubernetes:enabled:trueendpoint:https://localhost:6443kubeconfig:/var/lib/kubernetes/admin.kubeconfigkonnectivity:enabled:truemetrics_endpoint:http://localhost:8133# /metricshealth_endpoint:http://localhost:8092# /healthzetcd:enabled:truecacert:/var/lib/kubernetes/ca.pemcert:/var/lib/kubernetes/kubernetes.pemkey:/var/lib/kubernetes/kubernetes-key.pemendpoint:https://10.1.96.3:2379# /metricsnginx_vts:enabled:falseendpoint:http://localhost:9001# /metricsv_cdn_agent:enabled:falseendpoint:http://localhost:9093# /metricshaproxy:enabled:falseendpoint:http://localhost:8404# /metricsceph:enabled:falseendpoint:http://localhost:9283# /metricsv_dns:enabled:falseendpoint:http://localhost:9053# /metricssmart:enabled:falseblock_devices:# must exist, if not set, block devices are used from /sys/block/ (except for dmX and loopX)- /dev/sdakubernetes:pods:# v-agent must be running inside k8s for this to workenabled:falsenamespaces:- rook-ceph- defaultdcgm:# v-agent must be running inside k8s for this to workenabled:falsenamespace:gpu-operator # namespaceendpoint: nvidia-dcgm-exporter # name of the endpoint:k get endpoints

Currently, it's not 100% compatible with Kubernetes, that is to say running inside k8s and able to scrape k8s metrics. Right now it's largely an agent used to scrape metrics for services.

Building

Note: Agent must be built with cgo disabled, not doing so will result in GLIBC errors being thrown: CGO_ENABLED=0 go build -o v-agent cmd/v-agent/main.go