Write OTLP metrics by using an OpenTelemetry Collector sidecar

Setting up gcloud defaults

To configure gcloud with defaults for your Cloud Run service:

1. Set your default project:

   gcloudconfigsetprojectPROJECT_ID

   Replace PROJECT_ID with the name of the project you created for this tutorial.

2. Configure gcloud for your chosen region:

   gcloudconfigsetrun/regionREGION

   Replace REGION with the supported Cloud Run region of your choice.

Creating a Artifact Registry image repository

Create an Artifact Registry Docker repository to host the sample service image:

gcloud artifacts repositories create run-otel \
 --repository-format=docker \
 --location=REGION \
 --project=PROJECT_ID

Replace the following:

• PROJECT_ID with the name of the project you created for this tutorial.
• REGION REGION with the supported Cloud Run region of your choice.

Retrieving the code sample

To retrieve the code sample for use:

1. Clone the sample app repository to your local machine:

   Go

   gitclonehttps://github.com/GoogleCloudPlatform/golang-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

2. Change to the directory that contains the Cloud Run sample code:

   Go

   cdgolang-samples/run/custom-metrics/

Reviewing the code

The code for this tutorial consists of the following:

• A server that handles incoming requests and generates a metric named sidecar_sample_counter_total.

packagemain
import(
"context"
"fmt"
"log"
"net/http"
"os"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
"go.opentelemetry.io/otel/metric"
sdkmetric"go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/resource"
semconv"go.opentelemetry.io/otel/semconv/v1.24.0"
)
varcountermetric.Int64Counter
funcmain(){
ctx:=context.Background()
shutdown:=setupCounter(ctx)
defershutdown(ctx)
port:=os.Getenv("PORT")
ifport==""{
port="8080"
log.Printf("defaulting to port %s",port)
}
http.HandleFunc("/",handler)
log.Fatal(http.ListenAndServe(":"+port,nil))
}
funchandler(whttp.ResponseWriter,r*http.Request){
counter.Add(context.Background(),100)
fmt.Fprintln(w,"Incremented sidecar_sample_counter_total metric!")
}
funcsetupCounter(ctxcontext.Context)func(context.Context)error{
serviceName:=os.Getenv("K_SERVICE")
ifserviceName==""{
serviceName="sample-cloud-run-app"
}
r,err:=resource.Merge(
resource.Default(),
resource.NewWithAttributes(
resource.Default().SchemaURL(),
semconv.ServiceName(serviceName),
),
)
iferr!=nil{
log.Fatalf("Error creating resource: %v",err)
}
exporter,err:=otlpmetricgrpc.New(ctx,
otlpmetricgrpc.WithInsecure(),
)
iferr!=nil{
log.Fatalf("Error creating exporter: %s",err)
}
provider:=sdkmetric.NewMeterProvider(
sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter)),
sdkmetric.WithResource(r),
)
meter:=provider.Meter("example.com/metrics")
counter,err=meter.Int64Counter("sidecar-sample-counter")
iferr!=nil{
log.Fatalf("Error creating counter: %s",err)
}
returnprovider.Shutdown
}

• A Dockerfile that defines the operating environment for the service.

FROMgolang:1.24asbuilder
WORKDIR/app
COPY..
RUNCGO_ENABLED=0GOOS=linuxgobuild-osample-app
FROMalpine:3
RUNapkadd--no-cacheca-certificates
COPY--from=builder/app/sample-app/sample-app
CMD["/sample-app"]

The sample also includes files under the collector subdirectory for building a custom collector:

• A config file for the collector.

  receivers:
  otlp:
  protocols:
  grpc:
  http:
  processors:
  batch:
  #batchmetricsbeforesendingtoreduceAPIusage
  send_batch_max_size:200
  send_batch_size:200
  timeout:5s
  memory_limiter:
  #dropmetricsifmemoryusagegetstoohigh
  check_interval:1s
  limit_percentage:65
  spike_limit_percentage:20
  #automaticallydetectCloudRunresourcemetadata
  resourcedetection:
  detectors:[env,gcp]
  timeout:2s
  override:false
  resource:
  attributes:
  #addinstance_idasaresourceattribute
  -key:service.instance.id
  from_attribute:faas.id
  action:upsert
  #parseservicenamefromK_SERVICECloudRunvariable
  -key:service.name
  value:${env:K_SERVICE}
  action:insert
  exporters:
  googlemanagedprometheus:#Note:thisisintentionallyleftblank
  extensions:
  health_check:
  endpoint:0.0.0.0:13133
  service:
  extensions:[health_check]
  pipelines:
  metrics:
  receivers:[otlp]
  processors:[batch,memory_limiter,resourcedetection,resource]
  exporters:[googlemanagedprometheus]

• A Dockerfile that bundles the provided config into an upstream Collector image.

  FROMus-docker.pkg.dev/cloud-ops-agents-artifacts/google-cloud-opentelemetry-collector/otelcol-google:0.121.0
  COPYcollector-config.yaml/etc/otelcol-google/config.yaml

Shipping the code

Shipping code consists of three steps: building a container image with Cloud Build, uploading the container image to Artifact Registry, and deploying the container image to Cloud Run.

To ship your code:

1. Build your sample service container and publish on Artifact Registry:

   gcloudbuildssubmit--tagREGION-docker.pkg.dev/PROJECT_ID/run-otel/sample-metrics-app

   Upon success, you should see a SUCCESS message containing the ID, creation time, and image name. The image is stored in Artifact Registry and can be re-used if desired.

2. Build your Collector container and publish on Artifact Registry:

   gcloudbuildssubmitcollector--tagREGION-docker.pkg.dev/PROJECT_ID/run-otel/otel-collector-metrics

   Upon success, you should see a SUCCESS message containing the ID, creation time, and image name. The image is stored in Artifact Registry and can be re-used if desired.

3. Deploy your application:

   YAML

   1. Create a new file called service.yaml with the following:

      apiVersion: serving.knative.dev/v1
      kind: Service
      metadata:
       name: SERVICE-NAME
       annotations:
       run.googleapis.com/launch-stage: BETA
      spec:
       template:
       metadata:
       annotations:
       run.googleapis.com/container-dependencies: "{app:[collector]}"
       spec:
       containers:
       - image: REGION-docker.pkg.dev/PROJECT_ID/run-otel/sample-metrics-app
       name: app
       ports:
       - containerPort: CONTAINER_PORT
       env:
       - name: "OTEL_EXPORTER_OTLP_ENDPOINT"
       value: "http://localhost:4317"
       - image: REGION-docker.pkg.dev/PROJECT_ID/run-otel/otel-collector-metrics
       name: collector
       startupProbe:
       httpGet:
       path: /
       port: 13133
      

   2. Replace the following:
      • CONTAINER_PORT with the port your service's ingress container listens on. By default, this is 8080.
      • SERVICE-NAME with any name for your service, such as custom-metrics-sample-service.

4. Create the new service with the following command:

   gcloud run services replace service.yaml

   This command returns a service URL. Use this URL to try out the sample application in Trying it out.

Trying it out

Using the URL from the gcloud run command in Shipping the code, connect to the service to generate some sample metrics (you can run this command several times to generate more interesting data):

curl -H \
"Authorization: Bearer $(gcloud auth print-identity-token)" \
SERVICE_URL

Replace SERVICE_URL with the URL of your service.

Next, navigate to the Metrics Explorer within the Cloud Monitoring section of the Google Cloud console and select the sidecar_sample_counter_total metric.

Custom metric shown in the Metrics Explorer UI

You can also query the metrics with PromQL. For example, the query below will filter metrics based on the Cloud Run instance ID:

sidecar_sample_counter_total{instance="INSTANCE_ID"}

Replace INSTANCE_ID with the ID of any instance for your service (available in the instance logs or from the metadata server).

This query produces a chart like the one below:

Custom metric queried by PromQL