Machine Learning/LiftWing/KServe/DeployLocal
Note: This is a guide that installs kserve 0.10 in a local minikube kubernetes cluster that follows the instructions from the official documentation to install kserve in RawDeployment mode that doesn't need knative (and cannot scale down to zero).
Recommended Version Matrix
| Kubernetes Version | Recommended Istio Version |
|---|---|
| 1.22 | 1.11, 1.12 |
| 1.23 | 1.12, 1.13 |
| 1.24 | 1.13, 1.14 |
| 1.25 | 1.15, 1.16 |
1. Install minikube and start a cluster
Install minikube following the instructions from the official webpage.
Start a minikube cluster using k8s v1.23 (the memory and cpu arguments can be adjusted accordingly to suit the users requirements)
minikube--memory8192--cpus2--kubernetes-version=v1.23.16start
2. Install istio
curl-Lhttps://istio.io/downloadIstio|ISTIO_VERSION=1.13.0TARGET_ARCH=x86_64sh- cdistio-1.13.0 exportPATH=$PWD/bin:$PATH istioctlinstall
kubectlapply-f-<<EOF apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: istio spec: controller: istio.io/ingress-controller EOF
3. Install Cert Manager
kubectlapply-fhttps://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml
4. Install KServe
Install kserve CRDs.
kubectlapply-fhttps://github.com/kserve/kserve/releases/download/v0.10.0/kserve.yaml
Install kserve runtimes for prepackaged model servers (e.g. sklearn, torch, tensorflow etc.)
kubectlapply-fhttps://github.com/kserve/kserve/releases/download/v0.10.0/kserve-runtimes.yaml
We change the defaultDeployment in the inferenceservice-config to RawDeployment
kubectlpatchconfigmap/inferenceservice-config-nkserve--type=strategic-p'{"data": {"deploy": "{\"defaultDeploymentMode\": \"RawDeployment\"}"}}'
In the case that we use an IngressClass other than istio we need to change the IngressClassName in the inferenceservice-config to the corresponding name. In our case we use istio so there is no need to change anything.
5. Deploy first InferenceService and access our app
The easiest way to setup networking for our local cluster is to run the following command in a separate terminal:
minikubetunnel
This will allow us to communicate with our minikube cluster by acessing its load balancer through the external IP which will be set our localhost. To deploy an example InferenceService we use a pretrained sklearn model server.
kubectlapply-f-<<EOF apiVersion: "serving.kserve.io/v1beta1" kind: "InferenceService" metadata: name: "sklearn-iris" spec: predictor: model: args: ["--enable_docs_url=True"] modelFormat: name: sklearn storageUri: "gs://kfserving-examples/models/sklearn/1.0/model" EOF
Extract the SERVICE_HOSTNAME, INGRESS_HOST and INGRESS_PORT in order to communicate with the cluster
exportSERVICE_HOSTNAME=$(kubectlgetinferenceservicesklearn-iris-ojsonpath='{.status.url}'|cut-d"/"-f3) exportINGRESS_HOST=$(kubectl-nistio-systemgetserviceistio-ingressgateway-ojsonpath='{.status.loadBalancer.ingress[0].ip}') exportINGRESS_PORT=$(kubectl-nistio-systemgetserviceistio-ingressgateway-ojsonpath='{.spec.ports[?(@.name=="http2")].port}')
Create a file with input samples:
cat<<EOF > "./iris-input.json" { "instances": [ [6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6] ] } EOF
Extract the SERVICE_HOSTNAME and get your predictions using the REST endpoint:
curl-v-H"Host: ${SERVICE_HOSTNAME}"http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict-d@./iris-input.json
6. Install Minio (model storage)
Create a file called minio.yaml and install the minio test instance to your cluster:
kubectlapply-fminio.yaml
Install the Minio client (mc):
curl-LJ0https://dl.min.io/client/mc/release/linux-amd64/mc>mc chmod+xmc mc--help
In a different terminal window, port-forward our minio test app:
# Run port forwarding command in a different terminal kubectlport-forward$(kubectlgetpod--selector="app=minio"--outputjsonpath='{.items[0].metadata.name}')9000:9000
Add our test instance and create a bucket for model storage:
mcconfighostaddmyminiohttp://127.0.0.1:9000miniominio123 mcmbmyminio/wmf-ml-models
Create a s3-secret.yaml for minio and attach it to a service account:
kubectlapply-fs3-secret.yaml
8. Deploy enwiki-goodfaith and run a prediction
Upload enwiki-goodfaith model binary file:
mccpmodel.binmyminio/wmf-ml-models/enwiki-goodfaith/
Create an enwiki-goodfaith.yaml and apply the yaml to deploy the InferenceService on KServe:
kubectlapply-fenwiki-goodfaith.yaml
Same as the Step 5, run minikube tunnel to setup networking for our local cluster, and export env vars of the SERVICE_HOSTNAME, INGRESS_HOST and INGRESS_PORT.
Create a file with an input sample:
cat<<EOF > "./input.json" { "rev_id": 1145145653 } EOF
Run a prediction:
curl-v-H"Host: ${SERVICE_HOSTNAME}"http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/enwiki-goodfaith:predict-d@./input.json
Expected Output:
*Trying127.0.0.1:80... *Connectedto127.0.0.1(127.0.0.1)port80(#0) >POST/v1/models/enwiki-goodfaith:predictHTTP/1.1 >Host:enwiki-goodfaith-default.example.com >User-Agent:curl/7.86.0 >Accept:*/* >Content-Length:24 >Content-Type:application/x-www-form-urlencoded > *Markbundleasnotsupportingmultiuse <HTTP/1.1200OK <date:Fri,17Mar202315:21:50GMT <server:istio-envoy <content-length:194 <content-type:application/json <x-envoy-upstream-service-time:7055 < *Connection#0 to host 127.0.0.1 left intact {"enwiki":{"models":{"goodfaith":{"version":"0.5.1"}},"scores":{"1145145653":{"goodfaith":{"score":{"prediction":true,"probability":{"false":0.021491526258609506,"true":0.9785084737413905}}}}}}}%