The virtual device plugin for Kubernetes is a Daemonset that allows you to automatically:
- Expose arbitrary number of virtual GPUs on GPU nodes of your cluster.
- Run ML serving containers backed by Accelerator with low latency and low cost in your Kubernetes cluster.
This repository contains AWS virtual GPU implementation of the Kubernetes device plugin.
The list of prerequisites for running the virtual device plugin is described below:
- NVIDIA drivers ~= 361.93
- nvidia-docker version > 2.0 (see how to install and it's prerequisites)
- docker configured with nvidia as the default runtime.
- Kubernetes version >= 1.10
- This solution is build on top of Volta Multi-Process Service(MPS). You can only use it on instances types with Tesla-V100 or newer. (Only Amazon EC2 P3 Instances and Amazon EC2 G4 Instances now)
- Virtual GPU device plugin by default set GPU compute mode to
EXCLUSIVE_PROCESSwhich means GPU is assigned to MPS process, individual process threads can submit work to GPU concurrently via MPS server. This GPU can not be used for other purpose. - Virtual GPU device plugin only on single physical GPU instance like P3.2xlarge if you request
k8s.amazonaws.com/vgpumore than 1 in the workloads. - Virtual GPU device plugin can not work with Nvidia device plugin together. You can label nodes and use selector to install Virtual GPU device plugin.
kubectl label node <your_k8s_node_name> k8s.amazonaws.com/accelerator=vgpu
Update node selector label in the manifest file to match with labels of your GPU node group, then apply it to Kubernetes.
$ kubectl create -f https://raw.githubusercontent.com/awslabs/aws-virtual-gpu-device-plugin/v0.1.1/manifests/device-plugin.yml
Virtual NVIDIA GPUs can now be consumed via container level resource requirements using the resource name k8s.amazonaws.com/vgpu:
apiVersion: apps/v1 kind: Deployment metadata: name: resnet-deployment spec: replicas: 3 selector: matchLabels: app: resnet-server template: metadata: labels: app: resnet-server spec: # hostIPC is required for MPS communication hostIPC: true containers: - name: resnet-container image: seedjeffwan/tensorflow-serving-gpu:resnet args: # Make sure you set limit based on the vGPU account to avoid tf-serving process occupy all the gpu memory - --per_process_gpu_memory_fraction=0.2 env: - name: MODEL_NAME value: resnet ports: - containerPort: 8501 # Use virtual gpu resource here resources: limits: k8s.amazonaws.com/vgpu: 1 volumeMounts: - name: nvidia-mps mountPath: /tmp/nvidia-mps volumes: - name: nvidia-mps hostPath: path: /tmp/nvidia-mps
WARNING: if you don't request GPUs when using the device plugin all the GPUs on the machine will be exposed inside your container.
Check the full example here
Please check Development for more details.
The project idea comes from @RenaudWasTaken comment in kubernetes/kubernetes#52757 and Alibaba’s solution from @cheyang GPU Sharing Scheduler Extender Now Supports Fine-Grained Kubernetes Clusters.
AWS:
- 28 Nov 2018 - Amazon Elastic Inference – GPU-Powered Deep Learning Inference Acceleration
- 2 Dec 2018 - Amazon Elastic Inference - Reduce Deep Learning inference costs by 75%
- 30 JUL 2019 - Running Amazon Elastic Inference Workloads on Amazon ECS
- 06 SEP 2019 - Optimizing TensorFlow model serving with Kubernetes and Amazon Elastic Inference
- 03 DEC 2019 - Introducing Amazon EC2 Inf1 Instances, high performance and the lowest cost machine learning inference in the cloud
Community:
- Nvidia Turing GPU Architecture
- Nvidia Tesla V100 GPU Architecture
- Is sharing GPU to multiple containers feasible?
- Fractional GPUs: Software-based Compute and Memory Bandwidth Reservation for GPUs
- GPU Sharing Scheduler Extender Now Supports Fine-Grained Kubernetes Clusters
- GPU Sharing for Machine Learning Workload on Kubernetes - Henry Zhang & Yang Yu, VMware
- Deep Learning inference cost optimization practice on Kubernetes - Tencent
- Gaia Scheduler: A Kubernetes-Based Scheduler Framework
This project is licensed under the Apache-2.0 License.