Preemptible TPUs

Preemptible TPUs cost much less than non-preemptible TPUs. The Cloud TPU service might preempt (shut down) these TPUs at any time, if it requires additional TPU resources.

Creating a preemptible TPU VM

gcloud

$gcloudcomputetpustpu-vmcreatedemo-tpu\
--zone=europe-west4-a\
--accelerator-type=v3-8\
--version=tpu-vm-tf-2.17.1-pjrt\
--preemptible

where:

demo-tpu is a name for the TPU.
accelerator-type is the type of TPU. For more information about supported accelerator types for each TPU version, see TPU versions.
version is the Cloud TPU software version to install.
preemptible allows Cloud TPU to preempt the TPU.

Pricing and quota for preemptible TPUs

Pricing for preemptible TPUs is significantly lower than for normal TPUs. For details, see the pricing page. You are not charged for TPUs if they are preempted in the first minute after you create them.

Quota for preemptible TPUs is generally higher, and is separate from the quota for normal TPUs. See the quota page.

Detecting if a TPU has been preempted

You use the following gcloud command to check whether the Cloud TPU service has preempted your TPU:

List your available TPUs:

gcloudcomputetpustpu-vmlist
--zone=us-central1-b

The output of the command displays the details of the TPUs created in your project. If a TPU has been preempted, the status changes from READY to PREEMPTED.

For example:

NAMEZONEACCELERATOR_TYPENETWORK_ENDPOINTNETWORKRANGESTATUS
demo-tpuus-central1-bv2-810.240.1.2:8470default10.240.1.0/29PREEMPTED

Designing your machine learning application to run on preemptible TPUs

Make sure your application is resilient to restarts of the VM and TPU, by saving model checkpoints regularly and by configuring your application to restore the most recent checkpoint on restart.