Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

KEDA Scaling Issue: Jobs Continuously Created Until Karpenter Finishes Node Provisioning #2519

Unanswered
davidt-gh asked this question in Q&A
Discussion options

Hi all,

I’m encountering an issue with KEDA autoscaling while integrating it with Selenium Grid and Karpenter for node provisioning.

Scenario

  • I’m running Selenium tests and want KEDA to create exactly 3 jobs when I request 3 sessions.
  • However, KEDA continuously creates new jobs every few seconds until Karpenter finishes provisioning new nodes.

This results in more jobs being created than the actual number of Selenium sessions required.

Current Setup

Here is part of my values.yaml configuration:

selenium-grid:
 chromeNode:
 scaledOptions:
 minReplicaCount: 0
 maxReplicaCount: 15
 pollingInterval: 10
 autoscaling:
 enabled: true
 scalingType: job

Issue
• KEDA seems to poll the conditions too frequently (pollingInterval: 10) and creates new jobs when nodes are not yet ready.
• This causes KEDA to overscale beyond the requested session count.

Goal

I want KEDA to:
1. Create exactly 3 jobs for 3 sessions (no more, no less).
2. Avoid continuously spawning jobs while Karpenter provisions new nodes.

Proposed Solution

I believe the following changes might help resolve the issue:

  1. Increase pollingInterval and add a cooldownPeriod to prevent rapid scaling:
pollingInterval: 30
cooldownPeriod: 300
  1. Limit maxReplicaCount to the exact number of sessions requested:
maxReplicaCount: 3
  1. Adjust KEDA triggers to scale based on exact pending Selenium sessions, possibly using Prometheus metrics.

Questions
1. How can I fine-tune KEDA’s behavior to strictly match the requested number of sessions without overscaling?
2. Are there best practices to integrate KEDA with Karpenter to ensure smooth node scaling and job creation?

Any guidance or insights would be greatly appreciated!

Thank you!

You must be logged in to vote

Replies: 1 comment 3 replies

Comment options

Thanks for your detailed description.
In case Until Karpenter finishes provisioning new nodes, pods scaled up and kept status as Pending only, right?

You must be logged in to vote
3 replies
Comment options

Yes.
But if I sent 3 jobs (in total), until karpenter provide new nodes, I have 8 jobs == 8 8 pods, but only 3 are running.
Then after few minutes all of them cool down and removed.

Comment options

In case new pods scaled up with status Pending for a while, and how will KEDA core behave, which I am not sure much. I will recheck and let you know if I can find the correct answer.
In case pods scaled up and running. However, it still having overate scales, this one I am still working on improving the scaler logic. Hopefully, the upcoming KDEA core 2.16.1 to be released will have those improvements.

Comment options

Thanks!
I've thought it will already implemented when I selected Job, which suppose to create relevant jobs per needed sessions.
When I ask for X session it should create X jobs, and they need to wait in Pending mode until K8s complete them. Creating new jobs have no value in here.. Just saying.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet

AltStyle によって変換されたページ (->オリジナル) /