-
Notifications
You must be signed in to change notification settings - Fork 876
Bucket storage retention policy #6223
Hi Everyone,
I am using cortex for the long term storage of prometheus with the gcs (google cloud storage) bucket as the backend storage. I wanted to know how to configure the retention period for the metrics in the bucket. For my use case i only want to retain the metrics from the last 15 days and delete the data older than that from the bucket. Can i configure this in the cortex or do i need to apply the policy at the bucket level?
All reactions
Hey @ssharma9125, sorry for the spam. For your question, it is recommended to set up Cortex to apply retention via compactor instead of using GCS bucket level retention policy.
The config in compactor is called compactor_blocks_retention_period.
Replies: 2 comments 4 replies
All reactions
Hey @yeya24, I'm here to help you out with any issues or questions you have. I'm a bot that can assist with bug fixes, answer your queries, and guide you on how to contribute. Let me know how I can assist you!
To configure a retention policy in Cortex to retain metrics for the last 15 days and delete older data, you should use the RetentionPeriod field in the runtime configuration. This allows you to set a per-tenant retention period. You can set the RetentionPeriod to 15 days for the specific tenant or globally if no tenant-specific configuration is needed. This is done through the runtime config, which is reloaded periodically.
Here's a brief outline of how you can configure it:
- Use the
overridesmechanism in the runtime configuration file to set theRetentionPeriodfor the desired tenant(s). - Set the
RetentionPeriodto 15 days (in seconds) for the tenant(s) you want to apply this policy to.
The retention logic will then automatically handle the deletion of blocks older than 15 days by comparing the retention value to the block maxTime.
This policy should be applied within Cortex using the runtime configuration, rather than at the Google Cloud Storage bucket level, as Cortex handles the retention logic internally [1] .
All reactions
-
👎 1
@dosu what's the right config name for RetentionPeriod?
All reactions
Hey @ssharma9125, sorry for the spam. For your question, it is recommended to set up Cortex to apply retention via compactor instead of using GCS bucket level retention policy.
The config in compactor is called compactor_blocks_retention_period.
All reactions
hi @yeya24 can we limit the put and push-interval operations for s3 bucket, is there any configuration for limiting for size of bucket