Machine Learning/LiftWing/gRPC

This page documents how gRPC is set up on Lift Wing, how to enable it for an inference service, and how to call a model server over gRPC.

Lift Wing inference services normally accept REST requests using the KServe v1 protocol via POST /v1/models/<model_name>:predict. KServe also implements the Open Inference Protocol (the v2 protocol), which is served over both REST (POST /v2/models/<model_name>/infer) and gRPC. gRPC travels over HTTP/2 and uses a fixed protobuf contract.

The first consumer of gRPC on LiftWing is the Hoarde caching service, which talks to our Article Topic model server over the KServe v2 gRPC protocol. The work to enable this was tracked in T423582 (model-server code) and T424049 (cluster and networking).

A gRPC request from a client inside the production network reaches the model server through the same chain as an HTTPS REST request:

client
 │ gRPC over HTTP/2, TLS, SNI/:authority = <model>.<namespace>.wikimedia.org
 ▼
LVS (inference.svc.<dc>.wmnet:30443 or inference.discovery.wmnet:30443)
 ▼
Istio ingress gateway ← terminates TLS, routes by SNI / Host / :authority
 ▼ (HTTP/1.1 and HTTP/2 share the same port)
knative-local-gateway
 ▼
queue-proxy sidecar ← port 8012 for HTTP/1.1, port 8013 for h2c (HTTP/2)
 ▼
kserve-container ← REST on :8080, gRPC on :8081

The two facts that make this work, and that took the most effort to get right, are:

1. One ingress port 30443 serves both protocols. Istio can serve HTTP/1.1 and HTTP/2 on the same listener, gRPC is just HTTP/2 with a protobuf body, so it works on the existing 30443 HTTPS port without a dedicated gRPC port. This means there is no need to expose an additional port at the ingress level, nor to add it to LVS.
2. Knative uses a separate queue-proxy port for HTTP/2. The queue-proxy sidecar that fronts every Knative pod listens on 8012 for HTTP/1.1 and on 8013 for h2c (HTTP/2). Existing services only ever needed 8012, so 8013 had to be added to the NetworkPolicy before gRPC traffic could reach the container.

KServe's ModelServer by default exposes both v1 and v2 protocol, and starts a gRPC server automatically alongside the REST server. No extra flags are required to start the gRPC server, but one can disable gRPC server by passing enable_grpc=False to the ModelServer constructor.

What the model code must do is handle a v2 InferRequest and return a v2 InferResponse. gRPC and REST v2 differ in how the payload is encoded, so the model has to detect the transport. See this patch to see the changes needed to enable the v2 protocol.

Below we list the changes enabling the gRPC networking on LiftWing cluster. All of the required changes are part of the operations/deployment-charts repository.

The kserve-inference NetworkPolicy only allowed ingress to the queue-proxy on port 8012 designated for the HTTP/1.1 traffic. Until 8013 (h2c) was added, gRPC requests reached the gateway, but timed out before reaching the container with the error below. It was fixed in this patch.

StatusCode.UNAVAILABLE
"upstream connect error or disconnect/reset before headers. reset reason: connection timeout"

For Knative to route nicely to gRPC, the predictor's container port must be named h2c (Knative's convention for cleartext HTTP/2), which can be verified by checking that the Service port has appProtocol: kubernetes.io/h2c. See an example change.

Knative revisions can only expose a single container port. Pointing the revision at the gRPC port means REST is no longer reachable through Knative, so we run separate deployments for REST and gRPC.

Knative needs to recognise HTTP/2 connections rather than assuming HTTP/1.1. Setting autodetect-http2: enabled in the Knative config causes pods to run with ENABLE_HTTP2_AUTO_DETECTION=true. See an example change.

No Istio or LVS changes are needed for the ingress itself. gRPC is HTTP/2 over TLS, and the existing knative-ingress-gateway HTTPS listener on 30443 already accepts it alongside HTTP/1.1 REST traffic.

Istio routes to the right InferenceService by SNI and authority pseudo-header.

The KServe gRPC service is inference.GRPCInferenceService and the method used for prediction is ModelInfer. The payload convention on Lift Wing is a single BYTES input tensor named input whose contents are the UTF-8 JSON request body, which is mirroring the REST v2 data field.

First, run the service with docker-compose, it maps 8080 for REST and 8081 for gRPC. Next, you can use Python to send the request:

importgrpc
importjson
fromkserve.protocol.grpcimport grpc_predict_v2_pb2, grpc_predict_v2_pb2_grpc
channel = grpc.insecure_channel("localhost:8081")
request = grpc_predict_v2_pb2.ModelInferRequest()
request.model_name = "outlink-topic-model"
request.id = "test-123"
input_data = json.dumps({"page_id": 5355, "lang": "en"}).encode("utf-8")
tensor = request.inputs.add()
tensor.name = "input"
tensor.shape.extend([1])
tensor.datatype = "BYTES"
tensor.contents.bytes_contents.append(input_data)
stub = grpc_predict_v2_pb2_grpc.GRPCInferenceServiceStub(channel)
response = stub.ModelInfer(request)
result = json.loads(response.outputs[0].contents.bytes_contents[0].decode("utf-8"))
print(json.dumps(result, indent=2))

On the cluster the request goes through the Istio ingress over TLS on port 30443, so the client must:

• use a secure channel with the WMF CA bundle (/etc/ssl/certs/ca-certificates.crt on a deployment/stat host)
• set the SNI and authority to the service hostname, <model_name>.<namespace>.wikimedia.org - in gRPC Python these are grpc.ssl_target_name_override and grpc.default_authority respectively
• send the host metadata header with the same value, so the ingress routes to the right InferenceService

Example of the Python code:

importgrpc
importjson
fromkserve.protocol.grpcimport grpc_predict_v2_pb2, grpc_predict_v2_pb2_grpc
TARGET = "inference.discovery.wmnet:30443"
AUTHORITY = "outlink-topic-model.articletopic-outlink.wikimedia.org"
with open("/etc/ssl/certs/ca-certificates.crt", "rb") as f:
 ca_bundle = f.read()
channel = grpc.secure_channel(
 TARGET,
 grpc.ssl_channel_credentials(root_certificates=ca_bundle),
 options=(
 ("grpc.default_authority", AUTHORITY), # :authority pseudo-header
 ("grpc.ssl_target_name_override", AUTHORITY), # SNI
 ),
)
request = grpc_predict_v2_pb2.ModelInferRequest()
request.model_name = "outlink-topic-model"
request.id = "test-123"
metadata = [("host", AUTHORITY)] # Host header for ingress routing
input_data = json.dumps(
 {"page_id": 39755715, "revision_id": 1235690033, "wiki_id": "enwiki"}
).encode("utf-8")
tensor = request.inputs.add()
tensor.name = "input"
tensor.shape.extend([1])
tensor.datatype = "BYTES"
tensor.contents.bytes_contents.append(input_data)
stub = grpc_predict_v2_pb2_grpc.GRPCInferenceServiceStub(channel)
response = stub.ModelInfer(request, metadata=metadata)
result = json.loads(response.outputs[0].contents.bytes_contents[0].decode("utf-8"))
print(json.dumps(result, indent=2))