Cortex 1.17.0 #5913

This release contains 168 contributions from 29 contributors. We also have 16 new contributors. Thank you all for the contributions!

Some notable changes release are:

• Experimental OTLP ingestion
• Experimental minimize spread token generator strategy on Ingester
• Advanced query scheduling with Query Priority
• ListRules API high availability by rule group replication and backup
• Various improvements on Store Gateway Index Cache
• mem-ballast-size-bytes flag has been marked as deprecated and not functional anymore
• -querier.ingester-streaming flag has been marked as deprecated and ingester streaming is always enabled now
• querier.iterators and querier.batch-iterators flags have been marked as deprecated and batch iterator is always enabled in Querier now

Cortex

• [CHANGE] Azure Storage: Upgraded objstore dependency and support Azure Workload Identity Authentication. Added connection_string to support authenticating via SAS token. Marked msi_resource config as deprecating. Upgrade objstore and change Azure bucket client #5645
• [CHANGE] Store Gateway: Add a new fastcache based inmemory index cache. Replace inmemory index cache to fastcache based implementation #5619
• [CHANGE] Index Cache: Multi level cache backfilling operation becomes async. Added -blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency and -blocks-storage.bucket-store.index-cache.multilevel.max-async-buffer-size configs and metric cortex_store_multilevel_index_cache_backfill_dropped_items_total for number of dropped items. Make multilevel cache backfilling async #5661
• [CHANGE] Ingester: Disable uploading compacted blocks and overlapping compaction in ingester. Disable upload compacted blocks and overlapping compaction in ingester #5735
• [CHANGE] Distributor: Count the number of rate-limited samples in distributor_samples_in_total. Count the number of rate-limited samples in distributor_samples_in_total #5714
• [CHANGE] Ruler: Remove cortex_ruler_write_requests_total, cortex_ruler_write_requests_failed_total, cortex_ruler_queries_total, cortex_ruler_queries_failed_total, and cortex_ruler_query_seconds_total metrics for the tenant when the ruler deletes the manager for the tenant. Remove user specific evaluation metrics when rule manager is removed #5772
• [CHANGE] Main: Mark mem-ballast-size-bytes flag as deprecated. Mark mem ballast flag as deprecated #5816
• [CHANGE] Querier: Mark -querier.ingester-streaming flag as deprecated. Now query ingester streaming is always enabled. Always turn on ingester query streaming and mark its flag as deprecated #5817
• [CHANGE] Compactor/Bucket Store: Added -blocks-storage.bucket-store.block-discovery-strategy to configure different block listing strategy. Reverted the current recursive block listing mechanism and use the strategy Concurrent as in 1.15. Configurable block listing strategy #5828
• [CHANGE] Compactor: Don't halt compactor when overlapped source blocks detected. update thanos to main to disable overlapping sources check #5854
• [CHANGE] S3 Bucket Client: Expose -blocks-storage.s3.send-content-md5 flag and set default checksum algorithm to MD5. Add S3 send_content_md5 config and default to use MD5 #5870
• [CHANGE] Querier: Mark querier.iterators and querier.batch-iterators flags as deprecated. Now querier always use batch iterators. Always enable Querier BatchIterators #5868
• [FEATURE] OTLP ingestion experimental. OTLP Ingestion #5813
• [FEATURE] Ingester: Add per-tenant new metric cortex_ingester_tsdb_data_replay_duration_seconds. Add cortex_ingester_tsdb_data_replay_duration_seconds metric #5477
• [FEATURE] Query Frontend/Scheduler: Add query priority support. Query priority #5605
• [FEATURE] Tracing: Add kuberesolver to resolve endpoints address with kubernetes:// prefix as Kubernetes service. otlpgrpc: turn on round_robin LB policy and kuberesolver for resolving k8s service endpoints #5731
• [FEATURE] Tracing: Add tracing.otel.round-robin flag to use round_robin gRPC client side LB policy for sending OTLP traces. otlpgrpc: turn on round_robin LB policy and kuberesolver for resolving k8s service endpoints #5731
• [FEATURE] Ruler: Add ruler.concurrent-evals-enabled flag to enable concurrent evaluation within a single rule group for independent rules. Maximum concurrency can be configured via ruler.max-concurrent-evals. Add concurrent evaluation for ruler #5766
• [FEATURE] Distributor Queryable: Experimental: Add config zone_results_quorum_metadata. When querying ingesters using metadata APIs such as label names and values, only results from quorum number of zones will be included and merged. Option to include quorum zone results for DistributorQueryable metadata API #5779
• [FEATURE] Storage Cache Clients: Add config set_async_circuit_breaker_config to utilize the circuit breaker pattern for dynamically thresholding asynchronous set operations. Implemented in both memcached and redis cache clients. support set async circuit breaker for memcached and redis client #5789
• [FEATURE] Ruler: Add experimental experimental.ruler.api-deduplicate-rules flag to remove duplicate rule groups from the Prometheus compatible rules API endpoint. Add experimental ruler.ring.replication-factor and ruler.ring.zone-awareness-enabled flags to configure rule group replication, but only the first ruler in the replicaset evaluates the rule group, the rest will just hold a copy as backup. Add experimental experimental.ruler.api-enable-rules-backup flag to configure rulers to send the rule group backups stored in the replicaset to handle events when a ruler is down during an API request to list rules. Allowing rule backup for rules API HA #5782
• [ENHANCEMENT] Store Gateway: Added -store-gateway.enabled-tenants and -store-gateway.disabled-tenants to explicitly enable or disable store-gateway for specific tenants. add/support enabled_tenants and disabled_tenants feature in storegateway #5638
• [ENHANCEMENT] Compactor: Add new compactor metric cortex_compactor_start_duration_seconds. Add compactor start duration in seconds metric #5683
• [ENHANCEMENT] Index Cache: Multi level cache adds config max_backfill_items to cap max items to backfill per async operation. Improve async processor handling enabled items, optimize code further #5686
• [ENHANCEMENT] Query Frontend: Log number of split queries in query stats log. Log split queries at query frontend #5703
• [ENHANCEMENT] Logging: Added new options for logging HTTP request headers: -server.log-request-headers enables logging HTTP request headers, -server.log-request-headers-exclude-list allows users to specify headers which should not be logged. Upgrade Prometheus and Thanos version to latest main #5744
• [ENHANCEMENT] Query Frontend/Scheduler: Time check in query priority now considers overall data select time window (including range selectors, modifiers and lookback delta). Time check in query priority now considers overall data select time window (including range selectors, modifiers and lookback delta) #5758
• [ENHANCEMENT] Querier: Added querier.store-gateway-query-stats-enabled to enable or disable store gateway query stats log. Add flag to disable store gateway query stats #5749
• [ENHANCEMENT] AlertManager: Retrying AlertManager Delete Silence on error Add retry on Delete silence #5794
• [ENHANCEMENT] Ingester: Add new ingester metric cortex_ingester_max_inflight_query_requests. Adding cortex_ingester_max_inflight_query_requests metric #5798
• [ENHANCEMENT] Query: Added query_storage_wall_time to Query Frontend and Ruler query stats log for wall time spent on fetching data from storage. Query evaluation is not included. add query storage wall time to query stats #5799
• [ENHANCEMENT] Query: Added additional max query length check at Query Frontend and Ruler. Added -querier.ignore-max-query-length flag to disable max query length check at Querier. Consider query select time in max query length check #5808
• [ENHANCEMENT] Querier: Add context error check when converting Metrics to SeriesSet for GetSeries on distributorQuerier. add context cancellation checks on GetSeries #5827
• [ENHANCEMENT] Ruler: Improve GetRules response time by refactoring mutexes and introducing a temporary rules cache in ruler/manager.go. Improve GetRules performance #5805
• [ENHANCEMENT] Querier: Add context error check when merging slices from ingesters for GetLabel operations. add context cancellation checks on merging GetLabel slices #5837
• [ENHANCEMENT] Ring: Add experimental -ingester.tokens-generator-strategy=minimize-spread flag to enable the new minimize spread token generator strategy. Implementing Spread Minimizing Token Generator #5855
• [ENHANCEMENT] Query Frontend: Ensure error response returned by Query Frontend follows Prometheus API error response format. Refactor query frontend to return prometheus error response #5811
• [ENHANCEMENT] Ring Status Page: Add Ownership Diff From Expected column in the ring table to indicate the extent to which the ownership of a specific ingester differs from the expected ownership. Fixing how ring ownership is calculated on the ring status page #5889
• [BUGFIX] Distributor: Do not use label with empty values for sharding Filter empty labels from sharding key #5717
• [BUGFIX] Query Frontend: queries with negative offset should check whether it is cacheable or not. Query Frontend: queries with negative offset should check whether it is cacheable or not #5719
• [BUGFIX] Redis Cache: pass cache_size config correctly. Fix passing of redis client cache size #5734
• [BUGFIX] Distributor: Shuffle-Sharding with IngestionTenantShardSize == 0, default sharding strategy should be used setting -distributor.ingestion-tenant-shard-size to 0 throws error #5189
• [BUGFIX] Cortex: Fix GRPC stream clients not honoring overrides for call options. Fix GRPC client not honouring call options #5797
• [BUGFIX] Ring DDB: Fix lifecycle for ring counting unhealthy pods as healthy. Update cache on CAS for DDB #5838
• [BUGFIX] Ring DDB: Fix region assignment. Send ddb region as awsCfg #5842

New Contributors

• @testwill made their first contribution in chore: pkg import only once #5644
• @dsabsay made their first contribution in Upgrade Alpine to 3.18 #5684
• @pawarpranav83 made their first contribution in Query Frontend: queries with negative offset should check whether it is cacheable or not #5719
• @Kramer0x0 made their first contribution in Add codeql workflow #5743
• @tesla59 made their first contribution in docs: update getting_started docs #5746
• @blorby made their first contribution in Update footer.html #5767
• @CharlieTLe made their first contribution in Fix typos in code and docs #5784
• @lekaf974 made their first contribution in Update docker image consul (deprecated) by hashicorp/consul #5793
• @euniceek made their first contribution in Add retry on Delete silence #5794
• @mustafain117 made their first contribution in Fix support for keep_firing_for field in alert rules #5823
• @availhang made their first contribution in chore: remove repetitive words #5826
• @erlan-z made their first contribution in add context cancellation checks on GetSeries #5827
• @yj-yoo made their first contribution in change the reflect package to an unsafe package #5775
• @kindknow made their first contribution in chore: fix function names in comment #5856
• @momantech made their first contribution in Fix function names #5863
• @till made their first contribution in Update(ADOPTERS.md): add "us" #5874

Full Changelog: v1.16.1...v1.17.0