Proposal: Automatically terminate idle Kubernetes Computing Units #6264

@yrenat Thanks for creating the discussion. Before we dive into the details, I wonder whether this problem is common in Kubernetes. If so, are there any existing solutions to this problem? Please do some research and report your findings.

@chenlica
Based on my findings, this resource-waste problem is common. There are several generic solutions on "system" level, where they have a global control of what/how many resources are needed, but they can seldom address our specific problem where we need to control a specific pod/CU. The following are some of my findings, regarding garbage resource collection.

1. HPA / custom metrics
   This is the native idea from Kubernetes. It automatically changes the replica count of a scalable Kubernetes workload, such as a Deployment, ReplicaSet, ReplicationController, or StatefulSet, based on metrics like CPU, memory, or custom/external metrics. In simpler words, it can automatically scale up/down the number of pods/resources invovled. However, a Texera CU is a pod-specific per-user application resource with a DB record, access control, resource settings, and a corresponding runtime Pod (not just a pod). Also, "idle" is another Texera-specific, idea, which cannot be natively well-described by Kubernetes. Besides, HPA cannot fully shutdown the system. It has to have at least one pod running to keep track of all the computation resources.
2. KEDA
   KEDA (Kubernetes Event-driven Autoscaling) is designed for event-driven autoscaling and can scale workloads down to zero when event sources become inactive. This is a good solution for queue/event-driven services. However, using KEDA directly would require Texera to expose CU activity as an external/custom metric or event source, which will create multi layers of interference with Texera. It also would not automatically update Texera’s CU database state unless Texera adds extra integration.
3. Knative Serving
   Knative supports scale-to-zero for HTTP request/traffic-driven services. This is useful for serverless HTTP workloads, but a lot of Texera resources/commands are not handled in HTTP format. Adopting Knative would be a larger architectural change and still may not capture Texera’s definition of CU idleness. Besides, Knative also does not focus on pod/CU level logistics.

To sum up, for the current issue, I think the most suitable first solution is to implement our tailored idle detection inside Texera’s Computing Unit Managing Service or a Texera-side scheduled cleanup component. The cleanup should use Texera’s own CU termination logic rather than directly deleting Pods, so that both the Kubernetes Pod and the Texera DB state remain consistent.

Hi @yrenat We need this feature and thank you for working on this task! I did some research before and concluded that Kubernetes does not natively support this feature. I agree we need to have our own way mechanism to terminate idle CUs.

What's missing in both this discussion post and your PR is how you defined what "idle" CU means and the architectural design of how computing unit manager monitors all the CUs. Can you be more specific on those parts? Please include some visual diagrams if you can. We need to agree on the design before reviewing the PR.

@yrenat Thanks for reporting the findings. I am wondering: do other systems have something similar to "CU"? If not, what do they use? If the "Computing Unit (CU)" concept in Texera unique, we want to know its pros and cons compared to other systems.

I think this is a natural idea, you don't want to allocate more resources in the cluster than you need. I think the concept in k8s that would let you handle this best would be an Operator (https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). Then, Texera itself could through its operator tell Kubernetes it no longer needs some set of pods. I assume Texera currently doesn't have an operator? I took a cursory look and didn't find an implementation of one.

@yrenat Thanks for the nice summary. Currently a Texera CU is implemented as a Kubernetes pod. For the long term we do want to keep the abstraction of a CU, and choose other implementations such as a cluster of nodes to run the parallel Amber engine.

For other systems you have investigated, did you find any of them using a K8S pod as a basic computing unit?

I support the idea of separating a "logical CU" from a "physical CU." In this discussion of the lifecycle of a "CU," it refers to a physical CU, which is currently a k8s pod.

I wonder for other systems that also use a k8s pod:

1. Do they have the need to kill an idle k8s pod?
2. If so, how do they kill such a pod?
3. How does the termination of a pod affect the "logical CU"?

Hi all, after investigating current solutions and several discussions with @chenlica, we concluded that our current CU implementation with Kubernetes is a reasonable/common approach. Kubernetes itself also does not seem to provide a convenient built-in way to handle the garbage collection we need.

@chenlica agrees with my current garbage-collection implementation, but pointed out one issue: users cannot decide how long a CU can stay idle before being removed. Right now, this timeout is fixed by developers and is the same for all CUs/users.

We agreed that users should be able to set this value themselves. However, this may require passing the user input from the frontend to the backend and using it dynamically for each CU, which I assume is not very straightforward.

I would like to hear from anyone who has worked on similar frontend-to-backend parameter passing in this module, or knows something that I can refer to. Thanks

@xuang7 @aglinxinyuan : Can you share some thoughts?

Hi All, a brief update about the topic:

1. To provide a better picture of my PR that is related to this discussion, I think I should explain one of the key variables that I was using. One of the main functional variable to decide which CU has been idle is last_active, which is first introduced in this PR. This PR aims to clean garbage workflow execution results after a configured TTL. It starts a recurring cleanup task that periodically finds expired executions and removes it. I use last_active in my code because a CU is considered occupied only if there is at least one workflow running in this CU. So, if all workflow inside the given CU is inactive, it is safe to say the whole CU is inactive. We can then clean it.
2. As for on-the-fly update of the CU parameters (CU_idle_time_before_removal in my PR, to be specific), I have tried the methods mentioned above, but it turns out that they don't fit my use case well. What we really need is a pathway at CU level, after the CU is created. Sadly I didn't find a corresponding pathway, and I guess it would be a moderate level task to build such a thing. So, for now, I think it's better to keep the PR as it is (i.e., provide a global CU_idle_time_before_removal that fits into all CUs)