Cortex 1.20.0 Release Notes

This release contains 371 contributions from 38 contributors. We also have 14 new contributors. Thank you all for the contributions!

Some notable changes in this release are:

• Prometheus Remote Write 2.0 Support: Experimental support for Prometheus Remote Write 2.0 protocol.
• Parquet Format Support: Experimental Parquet based block storage. A new parquet converter service to convert TSDB blocks to parquet and querier to query parquet files.
• Query Federation with Regex Tenant Resolver: Introduce experimental regex tenant resolver allowing regex patterns in X-Scope-OrgID header via -tenant-federation.regex-matcher-enabled flag
• gRPC Stream Push between Distributor and Ingester: Experimental feature to use gRPC stream connections for push requests.
• More Native Histogram Support: Out-of-order native histogram ingestion support, per-tenant native histogram ingestion config, native histogram active series metrics and limits
• Resource-Based Monitor and Limiter: ResourceMonitor to collect CPU and Heap usage for Cortex and ResourceBasedLimiter in Ingesters and StoreGateways to protect the service from incoming requests when hitting limits
• UTF-8 Name: UTF-8 name support via -name-validation-scheme flag

What's Changed

• [CHANGE] StoreGateway/Alertmanager: Add default 5s connection timeout on client. Set default conn timeout of 5 sec for storeGateway and alertmanager clients #6603
• [CHANGE] Ingester: Remove EnableNativeHistograms config flag and instead gate keep through new per-tenant limit at ingestion. Replace EnableNativeHistograms from TSDB config with PerTenant Limit #6718
• [CHANGE] Validate a tenantID when to use a single tenant resolver. Validate tenantID on single resolver #6727
• [CHANGE] Ring: Add zone label to ring_members metric. Add zone label to ring_members metric. #6900
• [FEATURE] Distributor: Add an experimental -distributor.otlp.enable-type-and-unit-labels flag to add __type__ and __unit__ labels for OTLP metrics. Add enable type and unit labels flag #6969
• [FEATURE] Distributor: Add an experimental -distributor.otlp.allow-delta-temporality flag to ingest delta temporality otlp metrics. OTLP: Add a AllowDeltaTemporality flag to ingest delta temporality me... #6934
• [FEATURE] Query Frontend: Add dynamic interval size for query splitting. This is enabled by configuring experimental flags querier.max-shards-per-query and/or querier.max-fetched-data-duration-per-query. The split interval size is dynamically increased to maintain a number of shards and total duration fetched below the configured values. Dynamic splitting by interval for range queries #6458
• [FEATURE] Querier/Ruler: Add query_partial_data and rules_partial_data limits to allow queries/rules to be evaluated with data from a single zone, if other zones are not available. Query Partial Data #6526
• [FEATURE] Update prometheus alertmanager version to v0.28.0 and add new integration msteamsv2, jira, and rocketchat. Upgrade alertmanager version to v0.28.0 #6590
• [FEATURE] Ingester/StoreGateway: Add ResourceMonitor module in Cortex, and add ResourceBasedLimiter in Ingesters and StoreGateways. Add ResourceMonitor module in Cortex, and add ResourceBasedLimiter in Ingesters and StoreGateways #6674
• [FEATURE] Support Prometheus remote write 2.0. Support remote write v2 by converting request #6330
• [FEATURE] Ingester: Support out-of-order native histogram ingestion. It is automatically enabled when -ingester.out-of-order-time-window > 0 and -blocks-storage.tsdb.enable-native-histograms=true. Add ooo native histogram ingestion #6626 Make ooo native histograms ingestion automatically enabled #6663
• [FEATURE] Ruler: Add support for percentage based sharding for rulers. Add percentage based sharding to rulers #6680
• [FEATURE] Ruler: Add support for group labels. Add support for rule group labels #6665
• [FEATURE] Query federation: Introduce a regex tenant resolver to allow regex in X-Scope-OrgID value. Introduce a regex tenant resolver #6713

• Add an experimental tenant-federation.regex-matcher-enabled flag. If it enabled, user can input regex to X-Scope-OrgId, the matched tenantIDs are automatically involved. The user discovery is based on scanning block storage, so new users can get queries after uploading a block (generally 2h).
• Add an experimental tenant-federation.user-sync-interval flag, it specifies how frequently to scan users. The scanned users are used to calculate matched tenantIDs.

• [FEATURE] Experimental Support Parquet format: Implement parquet converter service to convert a TSDB block into Parquet and Parquet Queryable. Implementing Parquet Converter #6716 Implementing Parquet Queryable with fallback #6743
• [FEATURE] Distributor/Ingester: Implemented experimental feature to use gRPC stream connection for push requests. This can be enabled by setting -distributor.use-stream-push=true. Implement stream connection for remote write #6580
• [FEATURE] Compactor: Add support for percentage based sharding for compactors. Percentage Based Sharding for Compactors #6738
• [FEATURE] Querier: Allow choosing PromQL engine via header X-PromQL-EngineType. Querier: Allow switching between engines via header #6777
• [FEATURE] Querier: Support for configuring query optimizers and enabling XFunctions in the Thanos engine. Querier: Support configuring optimizers and XFunctions #6873
• [FEATURE] Query Frontend: Add support /api/v1/format_query API for formatting queries. Add support for /api/v1/format_query API #6893
• [FEATURE] Query Frontend: Add support for /api/v1/parse_query API (experimental) to parse a PromQL expression and return it as a JSON-formatted AST (abstract syntax tree). Parse query #6978
• [ENHANCEMENT] Upgrade the Prometheus version to 3.6.0 and add a -name-validation-scheme flag to support UTF-8. Upgrade prometheus to v3.6.0 #7040 Support UTF-8 name #7056
• [ENHANCEMENT] Distributor: Emit an error with a 400 status code when empty labels are found before the relabelling or label dropping process. Validate empty labels in push #7052
• [ENHANCEMENT] Parquet Storage: Add support for additional sort columns during Parquet file generation feat(parquetconverter): add support for additional sort columns during Parquet file generation #7003
• [ENHANCEMENT] Modernizes the entire codebase by using go modernize tool. Modernize the entire codebase #7005
• [ENHANCEMENT] Overrides Exporter: Expose all fields that can be converted to float64. Also, the label value max_local_series_per_metric got renamed to max_series_per_metric, and max_local_series_per_user got renamed to max_series_per_user. Override exporter: expose all fields that can be converted to float64 #6979
• [ENHANCEMENT] Ingester: Add cortex_ingester_tsdb_wal_replay_unknown_refs_total and cortex_ingester_tsdb_wbl_replay_unknown_refs_total metrics to track unknown series references during wal/wbl replaying. Add tsdb metrics to track unknown series references during wal/wbl re... #6945
• [ENHANCEMENT] Distributor: Introduce a Protobuf model for Prometheus Remote Write 2.0 and a pool to improve performance. Add prw2 pool #6917
• [ENHANCEMENT] Ruler: Emit an error message when the rule synchronization fails. Emit error when the rule synchronization fails #6902
• [ENHANCEMENT] Querier: Support snappy and zstd response compression for -querier.response-compression flag. add zstd and snappy compression for query api #6848
• [ENHANCEMENT] Tenant Federation: Add a # of query result limit logic when the -tenant-federation.regex-matcher-enabled is enabled. Add a query result limit to regex matcher #6845
• [ENHANCEMENT] Query Frontend: Add a cortex_slow_queries_total metric to track # of slow queries per user. Add a cortex_slow_queries_total to track # of slow queries #6859
• [ENHANCEMENT] Query Frontend: Change to return 400 when the tenant resolving fail. Return bad request when the tenant resolving fail #6715
• [ENHANCEMENT] Querier: Support query parameters to metadata api (/api/v1/metadata) to allow user to limit metadata to return. Add a -ingester.return-all-metadata flag to make the metadata API run when the deployment. Please set this flag to false to use the metadata API with the limits later. Implement metadata api query params #6681 Add ingester.skip-metadata-limits flag #6744
• [ENHANCEMENT] Ingester: Add a cortex_ingester_active_native_histogram_series metric to track # of active NH series. Add cortex_ingester_active_native_histogram_series metric to track ac... #6695
• [ENHANCEMENT] Query Frontend: Add new limit -frontend.max-query-response-size for total query response size after decompression in query frontend. Enforce query response size limit after decompression in query-frontend #6607
• [ENHANCEMENT] Alertmanager: Add nflog and silences maintenance metrics. Add nflog and silences metrics #6659
• [ENHANCEMENT] Querier: limit label APIs to query only ingesters if start param is not specified. Query ingesters only for GetLabelNames and GetLabelValues if start time parameter is not specified #6618
• [ENHANCEMENT] Alertmanager: Add new limits -alertmanager.max-silences-count and -alertmanager.max-silences-size-bytes for limiting silences per tenant. Add for tenant silences limit #6605
• [ENHANCEMENT] Add compactor.auto-forget-delay for compactor to auto forget compactors after X minutes without heartbeat. Compactor becomes unhealthy in the ring when stopped during startup #6533
• [ENHANCEMENT] StoreGateway: Emit more histogram buckets on the cortex_querier_storegateway_refetches_per_query metric. Emit more histogram buckets for cortex-querier_storegateway_refetches... #6570
• [ENHANCEMENT] Querier: Apply bytes limiter to LabelNames and LabelValuesForLabelNames. Apply bytes limiter to LabelNames and LabelValuesForLabelNames #6568
• [ENHANCEMENT] Query Frontend: Add a too_many_tenants reason label value to cortex_rejected_queries_total metric to track the rejected query count due to the # of tenant limits. Add a too_many_tenants label to cortex_rejected_queries_total metric #6569
• [ENHANCEMENT] Alertmanager: Add receiver validations for msteamsv2 and rocketchat. Add receiver validation for msteamsv2 and rocketchat #6606
• [ENHANCEMENT] Query Frontend: Add a -frontend.enabled-ruler-query-stats flag to configure whether to report the query stats log for queries coming from the Ruler. Add -frontend.enabled-ruler-query-stats flag #6504
• [ENHANCEMENT] OTLP: Support otlp metadata ingestion. Support otlp metadata ingestion #6617
• [ENHANCEMENT] AlertManager: Add keep_instance_in_the_ring_on_shutdown and tokens_file_path configs for alertmanager ring. Add tokenFile and persisit on ring features for AM ring #6628
• [ENHANCEMENT] Querier: Add metric and enhanced logging for query partial data. Add metric and enhanced logging for query partial data #6676
• [ENHANCEMENT] Ingester: Push request should fail when label set is out of order Push request should fail when label set is out of order #6746
• [ENHANCEMENT] Querier: Add querier.ingester-query-max-attempts to retry on partial data. Add querier.ingester-query-max-attempts to retry on partial data. #6714
• [ENHANCEMENT] Distributor: Add min/max schema validation for Native Histogram. Min/Max Schema Validation for NativeHistograms #6766
• [ENHANCEMENT] Ingester: Handle runtime errors in query path Ingester: Handle runtime errors in query path #6769
• [ENHANCEMENT] Compactor: Support metadata caching bucket for Cleaner. Can be enabled via -compactor.cleaner-caching-bucket-enabled flag. Support caching bucket for cleaner #6778
• [ENHANCEMENT] Distributor: Add ingestion rate limit for Native Histogram. Add nativeHistogram IngestionRate limit #6794 and Bug fix: Return 429 response when Native Histogram samples are discarded by Native Histogram Ingestion Rate Limiter #6994
• [ENHANCEMENT] Ingester: Add active series limit specifically for Native Histogram. Add active series limit for native histogram samples #6796
• [ENHANCEMENT] Compactor, Store Gateway: Introduce user scanner strategy and user index. Refactor block storage User Scanner and introduce user index file #6780
• [ENHANCEMENT] Querier: Support chunks cache for parquet queryable. caching bucket for parquet chunks file #6805
• [ENHANCEMENT] Parquet Storage: Add some metrics for parquet blocks and converter. Add cleaner and converter metrics for parquet #6809 Add parquet convert failure, block delay and total block to convert metrics #6821
• [ENHANCEMENT] Compactor: Optimize cleaner run time. Optimize cleaner run time #6815
• [ENHANCEMENT] Parquet Storage: Allow percentage based dynamic shard size for Parquet Converter. Add dynamic shard size for parquet converter #6817
• [ENHANCEMENT] Query Frontend: Enhance the performance of the JSON codec. Enhance the performance of the frontend JSON codec #6816
• [ENHANCEMENT] Compactor: Emit partition metrics separate from cleaner job. Split compactor cleaner metrics #6827
• [ENHANCEMENT] Metadata Cache: Support inmemory and multi level cache backend. Unify chunks and metadata cache backend to bucket cache #6829
• [ENHANCEMENT] Store Gateway: Allow to ignore syncing blocks older than certain time using ignore_blocks_before. Allow store gateway to ignore syncing blocks older than certain time #6830
• [ENHANCEMENT] Distributor: Add native histograms max sample size bytes limit validation. Add native histogram max sample size limit validation #6834
• [ENHANCEMENT] Querier: Support caching parquet labels file in parquet queryable. add parquet labels cache #6835
• [ENHANCEMENT] Querier: Support query limits in parquet queryable. Add query limiter support to parquet querier #6870
• [ENHANCEMENT] Ingester: Add new metric cortex_ingester_push_errors_total to track reasons for ingester request failures. New metric for ingester errors #6901
• [ENHANCEMENT] Ring: Expose detailed_metrics_enabled for all rings. Default true. Expose DetailedMetricsEnabled for all ring configs #6926
• [ENHANCEMENT] Parquet Storage: Allow Parquet Queryable to disable fallback to Store Gateway. Add a config to allow disable fallback in Parquet Queryable #6920
• [ENHANCEMENT] Query Frontend: Add a format_query and parse_query labels value to the op label at cortex_query_frontend_queries_total metric. Add a format query label value to queries total metric #6925 Add parse query op label to queries total metric #6990
• [ENHANCEMENT] API: add request ID injection to context to enable tracking requests across downstream services. add request ID injection to context to enable tracking requests across services #6895
• [ENHANCEMENT] gRPC: Add gRPC Channelz monitoring. Update weaveworks common to bring channelz monitoring #6950
• [ENHANCEMENT] Upgrade build image and Go version to 1.24.6. Chore: update build image to use Go 1.24.6 #6970 Chore: use Go 1.24.6 build image #6976
• [ENHANCEMENT] Implement versioned transactions for writes to DynamoDB ring. Add versioning to DynamoDB KV store #6986
• [ENHANCEMENT] Add source metadata to requests(api vs ruler) Introduced RequestSource metadata to distinguish between API and ruler queries #6947
• [ENHANCEMENT] Add new metric cortex_discarded_series and cortex_discarded_series_per_labelset to track number of series that have a discarded sample. Metric for number of series with discarded samples #6995
• [ENHANCEMENT] Ingester: Add cortex_ingester_tsdb_head_stale_series metric to keep track of number of stale series on head. Ingester: Add head stale series metric #7071
• [ENHANCEMENT] Expose more Go runtime metrics. Expose more Go runtime metrics #7070
• [ENHANCEMENT] Distributor: Filter out label with empty value. filter out label with empty value in distributor #7069
• [BUGFIX] Ingester: Avoid error or early throttling when READONLY ingesters are present in the ring Extend ShuffleSharding on READONLY ingesters #6517
• [BUGFIX] Ingester: Fix labelset data race condition. Fix labelset race condition #6573
• [BUGFIX] Compactor: Cleaner should not put deletion marker for blocks with no-compact marker. Cleaner should not put deletion marker for blocks with no-compact marker #6576
• [BUGFIX] Compactor: Cleaner would delete bucket index when there is no block in bucket store. Cleaner would delete bucket index when there is no block in bucket store #6577
• [BUGFIX] Querier: Fix marshal native histogram with empty bucket when protobuf codec is enabled. fix native histogram empty bucket marshal panic #6595
• [BUGFIX] Query Frontend: Fix samples scanned and peak samples query stats when query hits results cache. retrieve peakSamples and processedSamples query stats from results_cache #6591
• [BUGFIX] Query Frontend: Fix panic caused by nil pointer dereference. Fix panic caused by nil pointer dereference in Query Frontend #6609
• [BUGFIX] Ingester: Add check to avoid query 5xx when closing tsdb. Add check to avoid query 5xx when closing tsdb #6616
• [BUGFIX] Querier: Fix panic when marshaling QueryResultRequest. Fix querier panic when marshaling QueryResultRequest #6601
• [BUGFIX] Ingester: Avoid resharding for query when restart readonly ingesters. Keep readonly state when stopping lifecycler #6642
• [BUGFIX] Query Frontend: Fix query frontend per user metrics clean up. fix query frontend per tenant metrics leak when cleaning up user labels #6698
• [BUGFIX] Add __markers__ tenant ID validation. Add __markers__ tenantID validation #6761
• [BUGFIX] Ring: Fix nil pointer exception when token is shared. Fix nil pointer exception in GenerateTokens #6768
• [BUGFIX] Fix race condition in active user. Fix race condition in active user #6773
• [BUGFIX] Ruler: Prevent counting 2xx and 4XX responses as failed writes. Ruler: Prevent counting 2xx responses as failed writes #6785
• [BUGFIX] Ingester: Allow shipper to skip corrupted blocks. Skip corrupt blocks on shipper #6786
• [BUGFIX] Compactor: Delete the prefix blocks_meta from the metadata fetcher metrics. Delete prefix blocks_meta for meta fetcher metrics #6832
• [BUGFIX] Store Gateway: Avoid race condition by deduplicating entries in bucket stores user scan. Bug fix: Avoid race condition by deduplicating entries in bucket stores user scan #6863
• [BUGFIX] Runtime-config: Change to check tenant limit validation when loading runtime config only for all, distributor, querier, and ruler targets. change to check tenant limit validation only for some targets #6880
• [BUGFIX] Distributor: Fix /distributor/all_user_stats api to work during rolling updates on ingesters. fix AllUserStats during rolling updates on ingester #7026
• [BUGFIX] Runtime-config: Fix panic when the runtime config is null. fix panic when the runtime config is null #7062
• [BUGFIX] Scheduler: Avoid all queriers reserved for prioritized requests. fix: avoid all queriers reserved for prioritized requests #7057
• [BUGFIX] Compactor: Avoid race condition which allow a grouper to not compact all partitions. Fix visit marker race condition #7082
• [BUGFIX] Fix bug where validating metric names uses the wrong validation logic. Distributor failing to accept valid metric names #7086
• [BUGFIX] Add alertmanager receiver validation for discord and email. Add receiver validation (discord and email) #7097

New Contributors

• @dankrzeminski32 made their first contribution in update go to 1.24 #6636
• @CodingFabian made their first contribution in Added a link to the Dash0 OTEL collector config #6662
• @PaurushGarg made their first contribution in Replace EnableNativeHistograms from TSDB config with PerTenant Limit #6718
• @Ajay-Satish-01 made their first contribution in fix(grafana-datasource): Grafana Datasource pointing to wrong service #6819
• @bogdan-st made their first contribution in Add proposal for user overrides API #6818
• @siddarth2810 made their first contribution in docs: Add warning for inmemory ha-tracker to clarify its usage #6891
• @rubywtl made their first contribution in Query Frontend For Logical Query Plan Support #6884
• @guytet made their first contribution in docs: Add Open-Xchange to ADOPTERS.md #6915
• @avleentwilio made their first contribution in Add Twilio to list of adopters #6960
• @EpiJunkie made their first contribution in Chore: update build image to use Go 1.24.6 #6970
• @srlobo made their first contribution in Add Cabify to the list of adopters #7007
• @vivekgarg20 made their first contribution in Add Swiggy to Adopters list #7023
• @Angith made their first contribution in feat(parquetconverter): add support for additional sort columns during Parquet file generation #7003
• @aclaygray made their first contribution in Remove references to max_series_per_query from docs #6889

Full Changelog: v1.19.1...v1.20.0