Downsampling of metric data after a certain period #1834
-
|
Hello everyone :) I'm using Grafana Mimir as Prometheus data backend for a few days now and I'm very pleased with its performance. Now I have a question about data downsampling, because I didn't find anything about it in the documentation. As far as I understand Mimir will store all data for an unlimited time as long as I don't configure a retention time. Is it possible to configure it, that after some time (for example a week) the raw data will be aggregated as (for example) one hour data points which will be represented by a min, max and average value and the corresponding raw data will be deleted? Would be interesting if something like this is possible, because I don't need the raw data of every metric for all time, but some trend metrics would be nice. Because for reports like "how many http requests hit the loadbalancer two years ago" I don't need every 10s data point 😄. Thanks in advance for your help :) Kind regards, |
Beta Was this translation helpful? Give feedback.
Replies: 9 comments 22 replies
-
|
Hello! No, down sampling is not currently supported. I'm not sure if there are plans to work on it in the future. Are you interested in this because long range queries are taking a long time? Or to save on storage costs? Or some other reason? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the fast answer :) My main interest is to reduce storage space/cost in the long run (and as a bonus faster queries, because of the reduced dataset 😄). |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your question. There are several problems with downsampling that need to be considered:
These are just random thoughts that we would need to take into consideration when designing downsampling feature. |
Beta Was this translation helpful? Give feedback.
-
|
Great write up @pstibrany. Another thing to consider is that in order to provide accurate query results we would have to keep multiple values for 1 downsampled period, like the min or max of a series samples for each downsampled period. There's still opportunity to reduce the number of samples with downsampling (and so reduce storage, and increase query performance) but it's not a reduction from N to 1 per downsampled period. |
Beta Was this translation helpful? Give feedback.
-
|
A block by default is two hours of metrics? Might one be able to achieve coarse downsampling of historical data by deleting, say, every other block? Would it be possible to decrease the block size to decrease the blast radius of a deletion? |
Beta Was this translation helpful? Give feedback.
-
|
I don't care the storage cost but care user experience for long time range query. The main goal of downsampling is providing an opportunity to get fast results for long range queries like months or years. |
Beta Was this translation helpful? Give feedback.
-
|
As discussed on slack, I would like to share my usecases As users, we would like to query a wide range of serie(s). The full resolution is not mandatory. However, when runnning a wide range query the response time impact the user experience. # cluster level configuration
compactor:
downsampling:
- 1d:1m # After 1d apply downsampling add keep 1 sample per minute
- 2d:5m
- 2w:1h
# runtime configuration
overrides:
tenant1:
downsampling:
- 1d:1m # After 1d apply downsampling add keep 1 sample per minute
- 5d:5m
- 4w:1hWe can even think of downsampling differently a subset of the series associated with a tenant based on a regex. overrides:
tenant1:
downsampling:
".*":
- 1d:1m # After 1d apply downsampling add keep 1 sample per minute
- 5d:5m
- 4w:1h
"cpu.*":
- 1d:1m # After 1d apply downsampling add keep 1 sample per minute
- 4w:1hIn some case, once downsampled, the full resolution might not been needed. ExamplesHigh frequency samplingEnd users want to sample at 1Hz but the full resolution is needed only for a short period of time. Capacity planningKeep low resolution data for capacity planning. Indeed, the more data we have in the past the more accurate the forecasting is. The use case could be several years (2 to 5 years). Prune full resolutionOnly keep the downsampled data after a pre-defined period of time Additional contextside noteThis would help users migrate from other backend without feature loss:
|
Beta Was this translation helpful? Give feedback.
-
|
Hello, I must choose a new metric storage and I would like to use downsampling. Is this planned to add this to Mimir or I must go with Thanos ? Tky. |
Beta Was this translation helpful? Give feedback.
-
@sepich can you give us a sense of what fraction of your costs to run Mimir are attributable to object storage if you're saying that the 10% number is way off? |
Beta Was this translation helpful? Give feedback.
-
|
We do not run Mimir because it does not support downsampling, we run Thanos. In Thanos you can ship blocks directly to object storage from prometheus (skipping remote-write). In our case most of blocks are being shipped this way, and thanos-receive is a smaller part. Now to compute costs: |
Beta Was this translation helpful? Give feedback.
-
|
To give another point of view. I'm running on OVHcloud (full disclosure this is where I work). So even with an eventual downsampling feature I'm still far away from the expected 10% of cost due to storage. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for that datapoint for Mimir, Wilfried. So for you the object storage costs are actually significantly smaller than 10% of TCO. Of course this number will grow as you grow your retention period beyond nine months, but even if you quadruple that to 36 months (3 years), then we'd expect roughly 100 euro/month. That would put you at slightly more than 10% of your total infra spend on Mimir. (Note that our ~10% estimate is based on the expectation of 13 months retention, which is what we default to). I'm not as familiar with how Thanos works so I can't chime in on why there is such a discrepancy relative to the Thanos based estimate from @sepich . |
Beta Was this translation helpful? Give feedback.
-
|
Depending on how I scale my cluster, the storage costs may remain around 10%. Both compute and storage cost grow with the consumption of Mimir's tenant. For exemple, we need to account for 1 more CPU core for every 300k additional active series, same goes for the distributor with the number of sample/sec. Following this logic we need to account for the rest of the compute. This could add up rather quickly. Second example, with the default retention of 13 months (reported by @09jvilla), if a tenant loads a dashboard for a 90d timerange 6 months ago (e.g: quarter to quarter comparison), we should expect a fair amount of queries which will be splitted and sharded. This weight down on the read path especially on the querier and the store-gateway and therefore the bill. |
Beta Was this translation helpful? Give feedback.
-
I reckon performance on multi-month/multi-year queries is more important than the storage cost. Object storage pricing is generally low compared to the compute pricing. On alternative to both tackle performance and cost is to have downsampling and different retention per series. This way you could keep the downsampled metrics longer than the raw metrics. This is not possible at the moment as all metrics of a tenant are kept for the same amount of time. |
Beta Was this translation helpful? Give feedback.
-
That's exactly what we need. In our case, we don't care about precises metrics over time, we need to see the trends. We would like to have high resolutions metrics for a short range like 1 month, medium resolution metrics after that for like 6 months, and low resolution metrics for several years. |
Beta Was this translation helpful? Give feedback.
-
|
Is it maybe worth opening an issue for this since there is appetite from the community? i too have this requirement and for the same reasons as those above ^^ |
Beta Was this translation helpful? Give feedback.
-
|
Awesome idea 🔥 |
Beta Was this translation helpful? Give feedback.
-
|
Wow, that's nice to share your tips. |
Beta Was this translation helpful? Give feedback.
-
|
Tested the federate approach and works well. It is also possible the do the recording in Prometheus and use a proxy to set the tenant based on metrics labels. |
Beta Was this translation helpful? Give feedback.
-
|
@adirmatzkin Amazing idea. I have one question about it. Even though I scrape the metrics from the target with a higher scrape interval, the metric name remains the same. So how would we separate that the low-resolution metrics have to be sent to a different tenant when the metric names are the same? Wouldn't it send the same metrics to both tenants? |
Beta Was this translation helpful? Give feedback.
-
|
The point is to set the recording rule for a "long-term-storage tenant" to query the "original" tenant - that way the original and "downsampled" series won't be queried together when using them later. Have you seen @MrFreezeex's comment? |
Beta Was this translation helpful? Give feedback.
-
|
I've been thinking about a proposal for a long time. Here is what I've come up. Be advised it is not as easy as it looks. |
Beta Was this translation helpful? Give feedback.
Thank you for your question. There are several problems with downsampling that need to be considered:
when using object store like GCP or S3, storage costs are typically only small fraction of cost of running Mimir, so saving here may not be high.
downsampling is very IO intensive. Because it's not possible to modify TSDB blocks in place, downsampling requires that blocks are downloaded, rebuilt from scratch with downsampled series (possibly only some of them, based on configuration) and then uploaded back. Old blocks must be deleted. All this processing adds to the cost.
downsampling complicates querying. PromQL query engine uses single look-back period when looking for samples. Ty…