Skip to content

Navigation Menu

Sign in
Sign up

Remove the deprecated -ruler.evaluation-delay-duration limit - #7792

Merged
CharlieTLe merged 3 commits into
cortexproject:master from
CharlieTLe:release-1.22-pr-d3-remove-ruler-evaluation-delay
Sep 4, 2026
Merged

Remove the deprecated -ruler.evaluation-delay-duration limit #7792
CharlieTLe merged 3 commits into
cortexproject:master from
CharlieTLe:release-1.22-pr-d3-remove-ruler-evaluation-delay

Conversation

@CharlieTLe

@CharlieTLe CharlieTLe commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Groundwork for v1.22.0. Refs #5922.

Important

Stacked on #7790 and #7791. Please merge those first; this branch contains their
commits, so the diff here shows all three until they land. Review the third commit only.

Deprecated in 1.18.0, and its own help text has said since then that it would be
removed in v1.19.0 — three minors overdue:

"Deprecated(use ruler.query-offset instead) and will be removed in v1.19.0: ..."

-ruler.query-offset / ruler_query_offset replaces it, and Overrides.RulerQueryOffset()
no longer has to take the higher of the two values — it returns RulerQueryOffset directly.

Kept as its own PR because it has the widest blast radius of the three removals.

⚠️ Upgrade note for operators

ruler_evaluation_delay_duration is a per-tenant limit, so it can appear in the runtime
config as well as on the command line. pkg/cortex/runtime_config.go decodes the runtime
config with SetStrict(true), so a leftover override there makes the whole runtime config
fail to load:

  • On startup it is fatal. runtimeconfig.Manager.starting() returns the load error
    (pkg/util/runtimeconfig/manager.go), the service enters Failed, and Cortex.Run()
    stops the service manager — the process exits with
    module failed module=runtime-config err="failed to load runtime config: load file: yaml: unmarshal errors: ... field ruler_evaluation_delay_duration not found in type validation.plain".
  • On an already-running process it is not fatal. loop() logs the reload error and
    keeps going, which pins the last known-good overrides in place and drops
    cortex_runtime_config_last_reload_successful to 0.

So the upgrade fails loudly if you restart with a stale file, and silently stops picking up
new overrides if the field is added under a running process. Before upgrading:

grep -r ruler_evaluation_delay_duration <your runtime config>

The release notes should call this out prominently.

Integration tests

Five tests in integration/ruler_test.go set -ruler.evaluation-delay-duration: "0" to
disable the delay. The ruler query offset already defaults to 0, so dropping the flag
preserves their intent.

Verification

go build ./..., go vet (including the integration build tags), make doc (generated
config reference and JSON schema regenerated), and ./pkg/util/validation/... ./pkg/ruler/... ./pkg/cortex/... all green.

dosubot[bot] reacted with rocket emoji
@dosubot dosubot Bot added component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. type/breaking type/chore Something that needs to be done; not a bug or a feature labels Aug 20, 2026
CharlieTLe force-pushed the release-1.22-pr-d3-remove-ruler-evaluation-delay branch from 4aaf71d to e3b7c6b Compare August 20, 2026 01:20
CharlieTLe requested review from a team and yeya24 and removed request for a team August 20, 2026 19:08 
...reaming settings
Unlike the flag-only removals, both of these have a YAML config option and one of them
changes behaviour, so they are split out here.
-<prefix>.fifocache.size was deprecated in 1.1.0, five years and twenty-one minors ago.
A cache configured only through it now starts with no capacity, because NewFifoCache no
longer copies it into MaxSizeItems.
-querier.ingester-metadata-streaming defaulted to true and its help text has promised
since 1.18.0 that the feature would be always on. The querier now always uses the
streaming metadata RPCs, so the non-streaming branches in distributor_queryable.go and
the three non-streaming methods on the local Distributor interface are gone. The tests
that parameterised over streaming on/off collapse accordingly.
Also removes the hidden ingester_streaming YAML field, which had no reader at all - it
was left behind when -querier.ingester-streaming was deprecated in 1.17.0. Because Cortex
decodes config with UnmarshalStrict, that field was still silently accepted until now.
Generated config docs and the JSON schema are regenerated.
Signed-off-by: Charlie Le <charlie_le@apple.com>
Deprecated in 1.18.0, and its own help text has said since then that it would be removed
in v1.19.0 - three minors overdue. -ruler.query-offset replaces it, and RulerQueryOffset
no longer has to take the higher of the two values.
Kept as its own change because it has the widest blast radius of the deprecation
removals: ruler_evaluation_delay_duration is a per-tenant limit, so it can appear in the
runtime config as well as on the command line. Strict decoding means a leftover override
there fails every reload rather than failing startup, which pins the last good overrides
and only shows up as cortex_runtime_config_last_reload_successful going to 0.
The integration tests set the flag to 0 to disable the delay; the ruler query offset
already defaults to 0, so dropping the flag preserves their intent.
Signed-off-by: Charlie Le <charlie_le@apple.com>
CharlieTLe force-pushed the release-1.22-pr-d3-remove-ruler-evaluation-delay branch from e3b7c6b to 4dbbf69 Compare September 1, 2026 22:12
CharlieTLe requested a review from a team as a code owner September 1, 2026 22:12
Comment thread CHANGELOG.md Outdated
## master / unreleased
* [ENHANCEMENT] Query Frontend: Log `X-Grafana-User` header in query stats, slow query, and query request logs when Grafana's `send_user_header` is enabled. #7799
* [FEATURE] Engine: Add `-querier.selector-batch-size` and `-ruler.selector-batch-size` flags to configure series batching in the Thanos promQL engine. 0 disables batching. #7763
* [CHANGE] Ruler: Remove the deprecated `-ruler.evaluation-delay-duration` flag and its `ruler_evaluation_delay_duration` per-tenant limit. Use `-ruler.query-offset` / `ruler_query_offset`, which no longer takes the higher of the two values. Cortex decodes the runtime config strictly, so a leftover `ruler_evaluation_delay_duration` override will not crash the process: it makes every runtime config reload fail, pinning the last good overrides and dropping `cortex_runtime_config_last_reload_successful` to 0. Run `grep -r ruler_evaluation_delay_duration` over your runtime configs before upgrading. #7792

@SungJin1212 SungJin1212 Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the runtime config manager fails to start if the deprecated ruler_evaluation_delay_duration still remains when the user upgrades the Cortex version.

CharlieTLe reacted with thumbs up emoji

@CharlieTLe CharlieTLe Sep 3, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, and my write-up was wrong — thanks for catching it.

runtimeconfig.Manager.starting() returns the load error, so on a fresh start the
runtime-config service goes to Failed, Cortex.Run()'s failure listener stops the
service manager, and the process exits:

module failed module=runtime-config err="failed to load runtime config: load file: yaml: unmarshal errors:
 line 3: field ruler_evaluation_delay_duration not found in type validation.plain"

The "keeps running on the last known-good config" behaviour only applies to loop(), i.e.
when the field is added to the file under an already-running process — there the reload just
logs and cortex_runtime_config_last_reload_successful drops to 0.

I've corrected the CHANGELOG entry and the PR description to say it's fatal at startup.

 

...rtup
Signed-off-by: Charlie Le <charlie_le@apple.com>
CharlieTLe merged commit 7859b48 into cortexproject:master Sep 4, 2026
108 of 110 checks passed
CharlieTLe added a commit to CharlieTLe/cortex that referenced this pull request Sep 11, 2026 
Get the unreleased section into the shape RELEASE.md asks for before the
release-1.22 cut, so operators reading the notes see the changes that affect them:
- Delete the stale duplicate of cortexproject#7375, which already shipped in 1.21.0 as cortexproject#7370.
- Re-sort into CHANGE -> FEATURE -> ENHANCEMENT -> BUGFIX.
- Correct the gRPC entry: the bump landed at v1.82.1, not v1.79.3.
- Fold follow-up PRs into the entry they belong to.
- Add three missing user-facing entries: cortexproject#7513, cortexproject#7514 and cortexproject#7559.
- Reclassify as CHANGE the entries that break existing configs or log consumers:
 the sign-key validation (cortexproject#7587), the Alertmanager per-tenant *_file rejections
 (cortexproject#7767, cortexproject#7768, now one entry) and the time_taken -> time_taken_ms rename (cortexproject#7649).
- Note the operator impact of the distroless base image (cortexproject#7637) and of the
 500 -> 499 reclassification (cortexproject#7717).
Rebased onto master, which added eight entries after this was first written.
They are curated the same way:
- Sorted into their categories: the deprecated flag removal (cortexproject#7790) and the
 max-exemplars deprecation (cortexproject#7793) under CHANGE, the X-Grafana-User query log
 (cortexproject#7799) under ENHANCEMENT, the YAML zero-value validation (cortexproject#7700) and the
 ReadPartitionedGroupInfo error handling (cortexproject#7766) under BUGFIX.
- Folded the Go toolchain bump (cortexproject#7807, cortexproject#7814) into the existing build image
 entry, which now reads 1.27.0 rather than carrying a second entry for it.
- Folded cortexproject#7745 into cortexproject#7698: both are the same wipe-on-transient-DNS-failure bug,
 cortexproject#7698 on the A record path and cortexproject#7745 on the SRV path.
- Folded cortexproject#7743 into cortexproject#7640: both are panics in the active request tracker's
 truncation of match[]/query values.
Rebased again onto master, which added eight more entries. Same treatment:
- Sorted into their categories: the evaluation-delay-duration removal (cortexproject#7792)
 and the fifocache/ingester-metadata-streaming removals (cortexproject#7791) under CHANGE,
 the parquet max-block-label-names limit (cortexproject#7625), the non-pointer
 HistogramBucket slice (cortexproject#7809) and the merge iterator BatchSize (cortexproject#7823) under
 ENHANCEMENT, and the CSV-list empty-string fix (cortexproject#7714) under BUGFIX.
- Folded the Thanos/promql-engine refresh (cortexproject#7788) into the existing upgrade
 entry, which already carries cortexproject#7691, cortexproject#7505 and cortexproject#7740.
Signed-off-by: Charlie Le <charlie_le@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SungJin1212 SungJin1212 SungJin1212 approved these changes
@yeya24 yeya24 Awaiting requested review from yeya24 yeya24 is a code owner automatically assigned from cortexproject/maintainers

Assignees

No one assigned

Labels

component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. size/XL type/breaking type/chore Something that needs to be done; not a bug or a feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /