[Question] About HA prometheus instances with remote write

[winterTTr]

I tried but it seems I don't find any documentation for the solution to handle HA prometheus instances with remote write, so I create this ticket to ask for if some thanos feature already support it well or there are some suggestion to enable my scenario with thanos components.

Simply speaking, I wish to enable below features at the same time:

1. HA prometheus instances: such as we already create 2 prometheus instance which collect the same target, so any one down will not fully miss the metrics data
2. Remote write: the remote write here don't have to be thanos receiver. The purpose is, my team has our own analysis system, so we wish to export part of prometheus data into it. The receiver has already done today, but we find stability problem when there is single prometheus. So we try to enable HA on prometheus level.
3. we wish the data from two HA prometheus can be write to remote and deduplicated to our own receiver

I know a typical case to enable HA + data deduplication is:

     <--scrape ---  Prometheus replica instance 0  + sidecar    <--- |
Target                                                         Thanos Querier  --> deduplicated data
     <--scrape ---  Prometheus replica instance 1  + sidecar     <----|

So how should I enable HA + remote write + deduplication?
Below is just pseudo imagination, feel free to change anything in this diagram and give me some suggestion

     <--scrape ---  Prometheus replica instance 0  + sidecar(?)    --- remote write -->
Target                                                            Thanos XXXX  --remote write again ?---> deduplicated data  ->  my team's receiver to export data to another system (already done)
      <--scrape ---  Prometheus replica instance 1  + sidecar(?)     --- remote write -->

[yeya24]

If you want to use HA prometheuses with remote write, this kind of deduplication is not supported in Thanos right now.

There is a pr to handle deduplicated data from HA prometheus using the same online deduplication algorithm used by querier #4239. However, this only works for Thanos compactor and the data should be stored in object storage.

If you'd like to use your own storage system for deduplication, unfortuantely I don't think there is an out-of-box solution. You have to implement yourself. A good reference might be https://blog.timescale.com/blog/promscale-0-4-drawing-inspiration-from-cortex-to-rewrite-support-for-prometheus-high-availability/?utm_source=timescaledb&utm_medium=social&utm_campaign=abl-launch-2021&utm_content=promscale-ha-blog-0-4.

[yeya24]

Btw I just converted this issue to a discussion as it is a question.