How Can Cortext Handle Data Gaps in Network Partition Scenarios in HA Cluster of Prometheuses? #5633

On Mon, Nov 6, 2023 at 1:08 AM aleskxyz ***@***.***> wrote: Hi, Cortext selects a leader from the cluster of HA Prometheus to retrieve samples. Imagine a network partition situation where each Prometheus can scrape data from some instances. With the current Cortext design, only samples from the elected Prometheus will be written to long-term storage, and samples from other Prometheuses will be discarded, resulting in gaps for samples that are scraped only by the standby Prometheus. Does Cortext have a solution for this, or can it handle this situation like Thanos, which deduplicates data at query time? Thanks. — Reply to this email directly, view it on GitHub <#5633>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6XK4DPC7WEQ4D4SARLFK3YDCSJVAVCNFSM6AAAAAA67E7VSCVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZVHAYTQMBQGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

On Mon, Nov 6, 2023 at 10:33 AM aleskxyz ***@***.***> wrote: Thanks for your reply! As I told above, we may see this inconsistency in case of network partition. Imagine we have 2 prometheus in 2 different racks that both of them are scraping all instances. when internal connection between 2 racks is disrupted, then the active prometheus cannot scrape resources in the other rack but the local prometheus of that rack is still working. Thanks — Reply to this email directly, view it on GitHub <#5633 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6XK4FQKIV477GPNERF4D3YDEUQTAVCNFSM6AAAAAA67E7VSCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TIOJQHEYTE> . You are receiving this because you commented.Message ID: ***@***.***>