Configure Multus with IPVLAN and Whereabouts

Standard

This document explains how to configure Pods in Google Kubernetes Engine (GKE) with multiple network interfaces by using the Multus CNI, the IPVLAN CNI plugin, and the Whereabouts IPAM plugin.

The IPVLAN CNI plugin provides Layer 2 connectivity for additional Pod interfaces, and the Whereabouts IPAM plugin dynamically assigns IP addresses to them.

This setup enables advanced networking configurations, such as separating control plane and data plane traffic for enhanced network isolation and segmentation.

This document is for Cloud architects and Networking specialists who design and architect the network for their organization. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE user roles and tasks.

Before reading this document, ensure that you are familiar with the following concepts:

Benefits of using Multus with IPVLAN

Configuring your Pods with multiple network interfaces by using this solution provides several key advantages. The primary use cases for configuring Multus with IPVLAN in Layer 2 mode are for network segmentation that requires Layer 2 adjacency:

Traffic isolation: isolate different types of traffic for enhanced security and performance. For example, you can separate sensitive management traffic from application data traffic.
Control and data plane separation: dedicate the primary network interface for control plane traffic while directing high-throughput data plane traffic through a secondary IPVLAN interface.
Layer 2 adjacency: fulfill requirements for applications that need direct Layer 2 connectivity between Pods on the same secondary network.

Limitations

Pods configured with Multus interfaces cannot simultaneously use GKE's built-in multi-networking capabilities. A Pod's network configuration must use either Multus or the cluster's built-in multi-networking.

How Multus works with IPVLAN and Whereabouts

Multus is a CNI meta-plugin that enables Pods to attach to multiple networks. Multus acts as a dispatcher, calling other CNI plugins to configure network interfaces based on NetworkAttachmentDefinition resources. You define each additional network by using a NetworkAttachmentDefinition, which specifies which CNI plugin (such as IPVLAN) and IPAM plugin (such as Whereabouts) to use for that network.

The following diagram illustrates the Multus architecture with IPVLAN and Whereabouts plugins.The Whereabouts plugin works with Multus and IPVLAN to handle IP address management (IPAM) for the Pods' additional network interfaces.

Diagram showing how Multus, IPVLAN, and Whereabouts work together in GKE.

Figure 1. Multus architecture with IPVLAN and Whereabouts plugins.

This diagram shows two nodes that each have one Pod. Each Pod has a primary interface and an additional interface. The two primary interfaces connect to a shared network interface card, and the two additional interfaces connect to a different shared network interface card.

When using Multus with IPVLAN and Whereabouts on GKE, Pods typically have the following interface configuration:

Primary interface (eth0): GKE Dataplane V2 manages this interface, providing default cluster connectivity.
Additional interfaces (net1, etc.): Multus manages these interfaces. Multus invokes the IPVLAN CNI plugin in Layer 2 mode for each NetworkAttachmentDefinition that you specify in a Pod's annotations. This configuration provides Layer 2 connectivity to a secondary VPC network.
IP Address Management (IPAM): you configure the Whereabouts IPAM plugin within the NetworkAttachmentDefinition. The Whereabouts IPAM plugin dynamically assigns IP addresses to the additional IPVLAN interfaces from a predefined range.

Pod scheduling with multiple networks

When you create a Pod and specify a NetworkAttachmentDefinition in its annotations, the GKE scheduler places the Pod only on a node that can satisfy the network requirements. The scheduler identifies nodes within a node pool that have the necessary secondary network interface configured. This node identification process ensures that the scheduler schedules the Pod on a node that can connect to the additional network and receive an IP address from the specified range.

The following sections guide you through configuring Multus with IPVLAN and Whereabouts plugins on your GKE cluster.

Before you begin

Before you start, make sure that you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.
Note: For existing gcloud CLI installations, make sure to set the compute/region property. If you use primarily zonal clusters, set the compute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

Install the kubectl command-line tool.
Set up a GKE cluster running version 1.28 or later with Dataplane V2, IP Alias, and multi-networking enabled. To learn how, see Set up multi-network support for Pods. Enabling multi-networking also enables the Multi-IP-Subnet and Persistent-IP HA Policy features, which eliminate the need for manual inter-node connectivity setup.
Use a GKE-validated version of Multus CNI (such as v4.2.1) for compatibility.

Set up VPC

To set up the Virtual Private Cloud (VPC) to use with Multus, including creating a subnet for node networking and secondary ranges for Pod networking, complete the following steps:

Create a new VPC or use an existing one:
```
gcloudcomputenetworkscreateVPC_NAME\
--subnet-mode=custom
```
Replace VPC_NAME with the name of the VPC.
Create a new subnet within this VPC:
```
gcloudcomputenetworkssubnetscreateSUBNET_NAME\
--range=PRIMARY_RANGE\
--network=VPC_NAME\
--region=REGION\
--secondary-range=SECONDARY_RANGE_NAME=SECONDARY_RANGE_CIDR
```
Replace the following:
- SUBNET_NAME: the name of the new subnet.
- PRIMARY_RANGE: the primary CIDR range for the subnet, such as 10.0.1.0/24. This command uses this range for node interfaces.
- VPC_NAME: the name of the VPC.
- REGION: the region for the subnet, such as us-central1.
- SECONDARY_RANGE_NAME: the name of the secondary IP address range for Pods in the subnet.
- SECONDARY_RANGE_CIDR: the secondary CIDR range for Pods, such as 172.16.1.0/24. Additional interfaces on Pods use this range.
This command creates a subnet with a primary CIDR range for an additional node interface and a secondary range for the additional Pod interfaces.

Create a GKE Standard cluster

Create a GKE Standard cluster with multi-networking enabled:

gcloudcontainerclusterscreateCLUSTER_NAME\
--cluster-version=CLUSTER_VERSION\
--enable-dataplane-v2\
--enable-ip-alias\
--enable-multi-networking

Replace the following:

CLUSTER_NAME: the name of the new cluster.
CLUSTER_VERSION: the version of your GKE cluster. You must use version 1.28 or later.

Enabling multi-networking lets you create node pools with multiple network interfaces, which Multus CNI requires.

Create a GKE Standard node pool

Create a GKE Standard node pool connected to additional VPC networks:

gcloudcontainernode-poolscreateNODEPOOL_NAME\
--clusterCLUSTER_NAME\
--zone"ZONE"\
--additional-node-networknetwork=VPC_NAME,subnetwork=SUBNET_NAME\
--additional-pod-networksubnetwork=SUBNET_NAME,pod-ipv4-range=SECONDARY_RANGE_NAME,max-pods-per-node=8

Replace the following:

NODEPOOL_NAME: the name of the new node pool.
CLUSTER_NAME: the name of your cluster.
ZONE: the zone for the node pool, such as us-central1-c.
VPC_NAME: the name of the additional VPC.
SUBNET_NAME: the name of the subnet.
SECONDARY_RANGE_NAME: the name of the secondary IP address range for Pods in the subnet.

This command creates a node pool where nodes have an additional network interface in SUBNET_NAME, and Pods on these nodes can use IP addresses from SECONDARY_RANGE_NAME.

For more information about creating GKE clusters with multi-networking capabilities, see Set up multi-network support for pods.

Apply the Multus deployment

To enable multiple network interfaces for your Pods, install the Multus CNI plugin. Save the following manifest, which includes the required DaemonSet and Custom Resource Definition (CRD), as multus-manifest.yaml:

apiVersion:apiextensions.k8s.io/v1
kind:CustomResourceDefinition
metadata:
name:ippools.whereabouts.cni.cncf.io
spec:
group:whereabouts.cni.cncf.io
names:
kind:IPPool
listKind:IPPoolList
plural:ippools
singular:ippool
scope:Namespaced
versions:
-name:v1alpha1
schema:
openAPIV3Schema:
description:IPPool is the Schema for the ippools API
properties:
apiVersion:
description:'APIVersiondefinestheversionedschemaofthisrepresentation
ofanobject.Serversshouldconvertrecognizedschemastothelatest
internalvalue,andmayrejectunrecognizedvalues.Moreinfo:https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
type:string
kind:
description:'KindisastringvaluerepresentingtheRESTresourcethis
objectrepresents.Serversmayinferthisfromtheendpointtheclient
submitsrequeststo.Cannotbeupdated.InCamelCase.Moreinfo:https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
type:string
metadata:
type:object
spec:
description:IPPoolSpec defines the desired state of IPPool
properties:
allocations:
additionalProperties:
description:IPAllocation represents metadata about the pod/container
owner of a specific IP
properties:
id:
type:string
podref:
type:string
required:
-id
type:object
description:Allocations is the set of allocated IPs for the given
range. Its indices are a direct mapping to the IP with the same
index/offset for the pools range.
type:object
range:
description:Range is a RFC 4632/4291-style string that represents
an IP address and prefix length in CIDR notation
type:string
required:
-allocations
-range
type:object
type:object
served:true
storage:true
status:
acceptedNames:
kind:""
plural:""
conditions:[]
storedVersions:[]
---
apiVersion:apiextensions.k8s.io/v1
kind:CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version:v0.4.1
name:overlappingrangeipreservations.whereabouts.cni.cncf.io
spec:
group:whereabouts.cni.cncf.io
names:
kind:OverlappingRangeIPReservation
listKind:OverlappingRangeIPReservationList
plural:overlappingrangeipreservations
singular:overlappingrangeipreservation
scope:Namespaced
versions:
-name:v1alpha1
schema:
openAPIV3Schema:
description:OverlappingRangeIPReservation is the Schema for the OverlappingRangeIPReservations
API
properties:
apiVersion:
description:'APIVersiondefinestheversionedschemaofthisrepresentationofanobject.Serversshouldconvertrecognizedschemastothelatestinternalvalue,andmayrejectunrecognizedvalues.Moreinfo:https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type:string
kind:
description:'KindisastringvaluerepresentingtheRESTresourcethisobjectrepresents.Serversmayinferthisfromtheendpointtheclientsubmitsrequeststo.Cannotbeupdated.InCamelCase.Moreinfo:https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type:string
metadata:
type:object
spec:
description:OverlappingRangeIPReservationSpec defines the desired state
of OverlappingRangeIPReservation
properties:
containerid:
type:string
podref:
type:string
required:
-containerid
type:object
required:
-spec
type:object
served:true
storage:true
status:
acceptedNames:
kind:""
plural:""
conditions:[]
storedVersions:[]
---
kind:ConfigMap
apiVersion:v1
metadata:
name:multus-cni-config
namespace:kube-system
labels:
app:gke-multinet
data:
cni-conf.json:|
{
"name": "multus-cni-network",
"type": "multus",
"confDir": "/etc/cni/net.d",
"namespaceIsolation": true,
"logLevel": "verbose",
"logFile": "/var/log/multus.log",
"kubeconfig": "/var/lib/kubelet/kubeconfig",
"clusterNetwork": "gke-pod-network"
}
---
apiVersion:apiextensions.k8s.io/v1
kind:CustomResourceDefinition
metadata:
name:network-attachment-definitions.k8s.cni.cncf.io
spec:
group:k8s.cni.cncf.io
scope:Namespaced
names:
plural:network-attachment-definitions
singular:network-attachment-definition
kind:NetworkAttachmentDefinition
shortNames:
-net-attach-def
versions:
-name:v1
served:true
storage:true
schema:
openAPIV3Schema:
description:'NetworkAttachmentDefinitionisaCRDschemaspecifiedbytheNetworkPlumbing
WorkingGrouptoexpresstheintentforattachingpodstooneormorelogicalorphysical
networks.Moreinformationavailableat:https://github.com/k8snetworkplumbingwg/multi-net-spec'
type:object
properties:
apiVersion:
description:'APIVersiondefinestheversionedschemaofthisrepresen
tationofanobject.Serversshouldconvertrecognizedschemastothe
latestinternalvalue,andmayrejectunrecognizedvalues.Moreinfo:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type:string
kind:
description:'KindisastringvaluerepresentingtheRESTresourcethis
objectrepresents.Serversmayinferthisfromtheendpointtheclient
submitsrequeststo.Cannotbeupdated.InCamelCase.Moreinfo:https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type:string
metadata:
type:object
spec:
description:'NetworkAttachmentDefinitionspecdefinesthedesiredstateofanetworkattachment'
type:object
properties:
config:
description:'NetworkAttachmentDefinitionconfigisaJSON-formattedCNIconfiguration'
type:string
---
apiVersion:rbac.authorization.k8s.io/v1
kind:ClusterRole
metadata:
name:multus-role
rules:
-apiGroups:["k8s.cni.cncf.io"]
resources:
-'*'
verbs:
-'*'
---
kind:ClusterRole
apiVersion:rbac.authorization.k8s.io/v1
metadata:
name:whereabouts
rules:
-apiGroups:
-whereabouts.cni.cncf.io
resources:
-ippools
-overlappingrangeipreservations
verbs:
-get
-list
-watch
-create
-update
-patch
-delete
-apiGroups:
-coordination.k8s.io
resources:
-leases
verbs:
-create
-apiGroups:
-coordination.k8s.io
resources:
-leases
resourceNames:
-whereabouts
verbs:
-'*'
-apiGroups:[""]
resources:
-pods
verbs:
-list
-get
---
kind:ClusterRoleBinding
apiVersion:rbac.authorization.k8s.io/v1
metadata:
name:multus-role-binding
subjects:
-kind:Group
name:system:nodes
roleRef:
kind:ClusterRole
name:multus-role
apiGroup:rbac.authorization.k8s.io
---
apiVersion:rbac.authorization.k8s.io/v1
kind:ClusterRoleBinding
metadata:
name:whereabouts-role-binding
roleRef:
apiGroup:rbac.authorization.k8s.io
kind:ClusterRole
name:whereabouts
subjects:
-kind:ServiceAccount
name:whereabouts-sa
namespace:kube-system
---
apiVersion:v1
kind:ServiceAccount
metadata:
name:whereabouts-sa
namespace:kube-system
---
apiVersion:apps/v1
kind:DaemonSet
metadata:
name:gke-multinet
namespace:kube-system
labels:
app:gke-multinet
spec:
selector:
matchLabels:
app:gke-multinet
template:
metadata:
labels:
app:gke-multinet
spec:
priorityClassName:system-node-critical
hostNetwork:true
tolerations:
-operator:Exists
serviceAccountName:whereabouts-sa
containers:
-name:whereabouts-gc
command:[/ip-control-loop]
args:
-"--log-level=debug"
-"--enable-pod-watch=false"
-"--cron-schedule=*****"
image:gcr.io/gke-release/whereabouts:v0.7.0-gke.3@sha256:2bb8450a99d86c73b262f5ccd8c433d3e3abf17d36ee5c3bf1056a1fe479e8c2
env:
-name:NODENAME
valueFrom:
fieldRef:
apiVersion:v1
fieldPath:spec.nodeName
-name:WHEREABOUTS_NAMESPACE
valueFrom:
fieldRef:
fieldPath:metadata.namespace
resources:
requests:
cpu:"100m"
memory:"50Mi"
limits:
cpu:"100m"
memory:"50Mi"
initContainers:
-name:install-multus-config
image:gcr.io/gke-release/multus-cni:v4.2.1-gke.6@sha256:25b48b8dbbf6c78a10452836f52dee456514783565b70633a168a39e6d322310
args:
-"--cni-conf-dir=/host/etc/cni/net.d"
-"--multus-conf-file=/tmp/multus-conf/00-multus.conf"
-"--multus-log-level=verbose"
-"--multus-kubeconfig-file-host=/var/lib/kubelet/kubeconfig"
-"--skip-multus-binary-copy=true"
-"--skip-config-watch=true"
resources:
requests:
cpu:"100m"
memory:"50Mi"
limits:
cpu:"100m"
memory:"50Mi"
securityContext:
privileged:true
volumeMounts:
-name:cni
mountPath:/host/etc/cni/net.d
-name:multus-cfg
mountPath:/tmp/multus-conf
-name:install-whereabouts
command:["/bin/sh"]
args:
--c
->
SLEEP=false /install-cni.sh
image:gcr.io/gke-release/whereabouts:v0.7.0-gke.3@sha256:2bb8450a99d86c73b262f5ccd8c433d3e3abf17d36ee5c3bf1056a1fe479e8c2
env:
-name:NODENAME
valueFrom:
fieldRef:
apiVersion:v1
fieldPath:spec.nodeName
-name:WHEREABOUTS_NAMESPACE
valueFrom:
fieldRef:
fieldPath:metadata.namespace
resources:
requests:
cpu:"100m"
memory:"50Mi"
limits:
cpu:"100m"
memory:"50Mi"
securityContext:
privileged:true
volumeMounts:
-name:cni
mountPath:/host/etc/cni/net.d
-name:cnibin
mountPath:/host/opt/cni/bin
-name:install-binary
image:gcr.io/gke-release/multus-cni:v4.2.1-gke.6@sha256:25b48b8dbbf6c78a10452836f52dee456514783565b70633a168a39e6d322310
command:["/gkecmd"]
args:
-"-operation=copy"
-"-cni-bin-dir=/host/opt/cni/bin"
resources:
requests:
cpu:"10m"
memory:"100Mi"
limits:
cpu:"10m"
memory:"100Mi"
securityContext:
privileged:true
volumeMounts:
-name:cnibin
mountPath:/host/opt/cni/bin
volumes:
-hostPath:
path:/var/lib/kubelet/kubeconfig
type:File
name:kubelet-credentials
-name:cni
hostPath:
path:/etc/cni/net.d
type:DirectoryOrCreate
-name:cnibin
hostPath:
path:/home/kubernetes/bin
type:DirectoryOrCreate
-name:multus-cfg
configMap:
name:multus-cni-config
items:
-key:cni-conf.json
path:00-multus.conf
updateStrategy:
rollingUpdate:
maxUnavailable:2
type:RollingUpdate

Then, apply the manifest to your cluster:

kubectlapply-fmultus-manifest.yaml

Create a `NetworkAttachmentDefinition` manifest

To enable Pods to connect to additional networks, create a NetworkAttachmentDefinition manifest. This manifest defines how Pods connect to a network and specifies the IP address range that an IPAM plugin, such as Whereabouts, assigns. This range must be part of the subnet that connects your nodes' additional network interfaces.

Save this manifest as nad.yaml. This manifest uses IPVLAN and Whereabouts plugins.
```
apiVersion:"k8s.cni.cncf.io/v1"
kind:NetworkAttachmentDefinition
metadata:
name:NAD_NAME
spec:
config:'{
"cniVersion":"0.3.1",
"plugins":[
{
"type":"ipvlan",
"master":"eth1",
"mode":"l2",
"ipam":{
"type":"whereabouts",
"range":SECONDARY_RANGE_NAME
}
}
]
}'
```
The manifest includes the following fields:
- NAD_NAME: the name of your NetworkAttachmentDefinition.
- master: the name of the node's secondary network interface, which acts as the master interface for IPVLAN. On GKE, secondary network interfaces typically start with eth1 and are named sequentially. To confirm the interface name, connect to a node using SSH and run the ip addr command.
- range: he IP address range for Pod interfaces, which is the same as the secondary IPv4 range that you created for the Pods (SECONDARY_RANGE_NAME). For example, 172.16.1.0/24.
Apply the manifest to your cluster:
```
kubectlapply-fnad.yaml
```

Attach Pods to additional networks

To attach a Pod to an additional network, add the k8s.v1.cni.cncf.io/networks annotation to the Pod manifest. For multiple networks, provide a comma-separated list of NetworkAttachmentDefinition names in the following format: <namespace>/<nad-name>.

The following example shows a Pod manifest that attaches to the NetworkAttachmentDefinition named NAD_NAME in the default namespace:

apiVersion:v1
kind:Pod
metadata:
name:samplepod
annotations:
k8s.v1.cni.cncf.io/networks:default/NAD_NAME
spec:
containers:
-name:sample-container
image:nginx

Replace NAD_NAME with the name of the NetworkAttachmentDefinition you created.

When you apply this manifest, Kubernetes creates the Pod with an additional network interface (net1) connected to the network that the NetworkAttachmentDefinition specifies.

Verify the Pod's additional IP address

To verify that the Pod receives an additional IP address after you attach the Pod to an additional network, inspect the network interfaces within the Pod:

To inspect the samplepod and verify the additional IP address, use the following command:
```
$kubectldescribepodPODNAME
```
Replace the PODNAME with the name of your Pod, such as samplepod.

Examine the output. The eth0 interface has the Pod's primary IP address. The Whereabouts plugin assigns the additional IP address to another interface, such as net1.

The output is similar to the following:

k8s.v1.cni.cncf.io/network-status:
 [{
 "name": "gke-pod-network",
 "interface": "eth0",
 "ips": [
 "10.104.3.4"
 ],
 "mac": "ea:e2:f6:ce:18:b5",
 "default": true,
 "dns": {},
 "gateway": [
 "\u003cnil\u003e"
 ]
 },{
 "name": "default/my-nad",
 "interface": "net1",
 "ips": [
 "10.200.1.1"
 ],
 "mac": "42:01:64:c8:c8:07",
 "dns": {}
 }]
k8s.v1.cni.cncf.io/networks: default/my-nad

In this example, 10.104.5.19 is the primary IP address on eth0, and 10.200.1.1 is the additional IP address on net1.

What's next

Learn more about Multus CNI.
Learn more about IPVLAN CNI.
Learn more about Whereabouts IPAM.