Migrate GKE Inference Gateway from v1alpha2 to v1

This page explains how to migrate your GKE Inference Gateway setup from the preview v1alpha2 API to the generally available v1 API.

This document is intended for platform administrators and networking specialists who are using the v1alpha2 version of the GKE Inference Gateway and want to upgrade to the v1 version to use the latest features.

Before you start the migration, ensure you are familiar with the concepts and deployment of the GKE Inference Gateway. We recommend you review Deploy GKE Inference Gateway.

Before you begin

Before you start the migration, determine if you need to follow this guide.

Check for existing v1alpha2 APIs

To check if you're using the v1alpha2 GKE Inference Gateway API, run the following commands:

kubectlgetinferencepools.inference.networking.x-k8s.io--all-namespaces
kubectlgetinferencemodels.inference.networking.x-k8s.io--all-namespaces

The output of these commands determines if you need to migrate:

• If either command returns one or more InferencePool or InferenceModel resources, you are using the v1alpha2 API and must follow this guide.
• If both commands return No resources found, you are not using the v1alpha2 API. You can proceed with a fresh installation of the v1 GKE Inference Gateway.

Migration paths

There are two paths for migrating from v1alpha2 to v1:

• Simple migration (with downtime): this path is faster and simpler but results in a brief period of downtime. It is the recommended path if you don't require a zero-downtime migration.
• Zero-downtime migration: this path is for users who cannot afford any service interruption. It involves running both v1alpha2 and v1 stacks side-by-side and gradually shifting traffic.

Simple migration (with downtime)

This section describes how to perform a simple migration with downtime.

1. Delete existing v1alpha2 resources: to delete the v1alpha2 resources, choose one of the following options:

   Option 1: Uninstall using Helm

   helmuninstallHELM_PREVIEW_INFERENCEPOOL_NAME
   

   Option 2: Manually delete resources

   If you are not using Helm, manually delete all resources associated with your v1alpha2 deployment:

   • Update or delete the HTTPRoute to remove the backendRef that points to the v1alpha2 InferencePool.
   • Delete the v1alpha2 InferencePool, any InferenceModel resources that point to it, and the corresponding Endpoint Picker (EPP) Deployment and Service.

   After all v1alpha2 custom resources are deleted, remove the Custom Resource Definitions (CRD) from your cluster:

   kubectldelete-fhttps://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.3.0/manifests.yaml
   

2. Install v1 resources: after you clean up the old resources, install the v1 GKE Inference Gateway. This process involves the following:

   1. Install the new v1 Custom Resource Definitions (CRDs).
   2. Create a new v1 InferencePool and corresponding InferenceObjective resources. The InferenceObjective resource is still defined in the v1alpha2 API.
   3. Create a new HTTPRoute that directs traffic to your new v1 InferencePool.

3. Verify the deployment: after a few minutes, verify that your new v1 stack is correctly serving traffic.

   1. Confirm that the Gateway status is PROGRAMMED:

      kubectlgetgateway-owide
      

      The output should look similar to this:

      NAME CLASS ADDRESS PROGRAMMED AGE
      inference-gateway gke-l7-regional-external-managed <IP_ADDRESS> True 10m
      

   2. Verify the endpoint by sending a request:

      IP=$(kubectlgetgateway/inference-gateway-ojsonpath='{.status.addresses[0].value}')
      PORT=80
      curl-i${IP}:${PORT}/v1/completions-H'Content-Type: application/json'-d'{"model": "<var>YOUR_MODEL</var>","prompt": "<var>YOUR_PROMPT</var>","max_tokens": 100,"temperature": 0}'
      

   3. Ensure you receive a successful response with a 200 response code.

Zero-downtime migration

This migration path is designed for users who cannot afford any service interruption. The following diagram illustrates how GKE Inference Gateway facilitates serving multiple generative AI models, a key aspect of a zero-downtime migration strategy.

Distinguishing API versions with kubectl

During the zero-downtime migration, both v1alpha2 and v1 CRDs are installed on your cluster. This can create ambiguity when using kubectl to query for InferencePool resources. To ensure you are interacting with the correct version, you must use the full resource name:

• For v1alpha2:

  kubectlgetinferencepools.inference.networking.x-k8s.io
  

• For v1:

  kubectlgetinferencepools.inference.networking.k8s.io
  

The v1 API also provides a convenient short name, infpool, which you can use to query v1 resources specifically:

kubectlgetinfpool

Stage 1: Side-by-side v1 deployment

In this stage, you deploy the new v1 InferencePool stack alongside the existing v1alpha2 stack, which allows for a safe, gradual migration.

After you finish all the steps in this stage, you have the following infrastructure in the following diagram:

1. Install needed Custom Resource Definition (CRDs) in your GKE cluster:

   • For GKE versions earlier than 1.34.0-gke.1626000, run the following command to install both the v1 InferencePool and alpha InferenceObjective CRDs:

   kubectlapply-fhttps://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.0.0/manifests.yaml
   

   • For GKE versions 1.34.0-gke.1626000 or later, install only the alpha InferenceObjective CRD by running the following command:

   kubectlapply-fhttps://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/v1.0.0/config/crd/bases/inference.networking.x-k8s.io_inferenceobjectives.yaml
   

2. Install the v1 InferencePool.

   Use Helm to install a new v1 InferencePool with a distinct release name, such as vllm-llama3-8b-instruct-ga. The InferencePool must target the same Model Server pods as the alpha InferencePool using inferencePool.modelServers.matchLabels.app.

   To install the InferencePool, use the following command:

   helminstallvllm-llama3-8b-instruct-ga\
   --setinferencePool.modelServers.matchLabels.app=MODEL_SERVER_DEPLOYMENT_LABEL\
   --setprovider.name=gke\
   --versionRELEASE\
   oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool
   

3. Create v1alpha2 InferenceObjective resources.

   As part of migrating to the v1.0 release of Gateway API Inference Extension, we also need to migrate from the alpha InferenceModel API to the new InferenceObjective API.

   1. Apply the following YAML to create the InferenceObjective resources:

      kubectlapply-f-<<EOF
      ---
      apiVersion:inference.networking.x-k8s.io/v1alpha2
      kind:InferenceObjective
      metadata:
      name:food-review
      spec:
      priority:2
      poolRef:
      group:inference.networking.k8s.io
      name:vllm-llama3-8b-instruct-ga
      ---
      apiVersion:inference.networking.x-k8s.io/v1alpha2
      kind:InferenceObjective
      metadata:
      name:base-model
      spec:
      priority:2
      poolRef:
      group:inference.networking.k8s.io
      name:vllm-llama3-8b-instruct-ga
      ---
      EOF
      

Stage 2: Traffic shifting

With both stacks running, you can start shifting traffic from v1alpha2 to v1 by updating the HTTPRoute to split traffic. This example shows a 50-50 split.

1. Update HTTPRoute for traffic splitting.

   To update the HTTPRoute for traffic splitting, run the following command:

   kubectlapply-f-<<EOF
   ---
   apiVersion:gateway.networking.k8s.io/v1
   kind:HTTPRoute
   metadata:
   name:llm-route
   spec:
   parentRefs:
   -group:gateway.networking.k8s.io
   kind:Gateway
   name:inference-gateway
   rules:
   -backendRefs:
   -group:inference.networking.x-k8s.io
   kind:InferencePool
   name:vllm-llama3-8b-instruct-preview
   weight:50
   -group:inference.networking.k8s.io
   kind:InferencePool
   name:vllm-llama3-8b-instruct-ga
   weight:50
   ---
   EOF
   

2. Verify and monitor.

   After applying the changes, monitor the performance and stability of the new v1 stack. Verify that the inference-gateway gateway has a PROGRAMMED status of TRUE.

Stage 3: Finalization and cleanup

Once you have verified that the v1 InferencePool is stable, you can direct all traffic to it and decommission the old v1alpha2 resources.

1. Shift 100% of traffic to the v1 InferencePool.

   To shift 100 percent of traffic to the v1 InferencePool, run the following command:

   kubectlapply-f-<<EOF
   ---
   apiVersion:gateway.networking.k8s.io/v1
   kind:HTTPRoute
   metadata:
   name:llm-route
   spec:
   parentRefs:
   -group:gateway.networking.k8s.io
   kind:Gateway
   name:inference-gateway
   rules:
   -backendRefs:
   -group:inference.networking.k8s.io
   kind:InferencePool
   name:vllm-llama3-8b-instruct-ga
   weight:100
   ---
   EOF
   

2. Perform final verification.

   After directing all traffic to the v1 stack, verify that it is handling all traffic as expected.

   1. Confirm that the Gateway status is PROGRAMMED:

      kubectlgetgateway-owide
      

      The output should look similar to this:

      NAME CLASS ADDRESS PROGRAMMED AGE
      inference-gateway gke-l7-regional-external-managed <IP_ADDRESS> True 10m
      

   2. Verify the endpoint by sending a request:

      IP=$(kubectlgetgateway/inference-gateway-ojsonpath='{.status.addresses[0].value}')
      PORT=80
      curl-i${IP}:${PORT}/v1/completions-H'Content-Type: application/json'-d'{
      "model": "YOUR_MODEL,
      "prompt": YOUR_PROMPT,
      "max_tokens": 100,
      "temperature": 0
      }'
      

   3. Ensure you receive a successful response with a 200 response code.

3. Clean up v1alpha2 resources.

   After confirming the v1 stack is fully operational, safely remove the old v1alpha2 resources.

4. Check for remaining v1alpha2 resources.

   Now that you've migrated to the v1 InferencePool API, it's safe to delete the old CRDs. Check for existing v1alpha2 APIs to ensure you no longer have any v1alpha2 resources in use. If you still have some remaining, you can continue the migration process for those.

5. Delete v1alpha2 CRDs.

   After all v1alpha2 custom resources are deleted, remove the Custom Resource Definitions (CRD) from your cluster:

   kubectldelete-fhttps://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.3.0/manifests.yaml
   

   After completing all steps, your infrastructure should resemble the following diagram:

   Routing requests to different models based on model name and Priority

What's next

• Learn more about Deploy GKE Inference Gateway.
  • Explore other GKE networking features.