You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, I use cortex frontend only in my project, I only enabled query-fronend module with step alignment and splitting in yaml config and I also configured my own downstream prometeus url, below is my config file.
With my load test with k6 to my cortex frontend, I'm seeing most of my requests end-to-end duration is 60s, even though my downstream prometheus returned response in ~14s in P99. After adding tracing and debugging, I find out it's because this io.Copy(w, resp.body) taking time. I printed out error returned by this line, it returned this error: write tcp 10.200.1.15:9090->10.200.0.126:36774: i/o timeout. These requests didn't supply Accept-Encoding: gzip in request header.
After I supply Accept-Encoding: gzip to the request header, the i/o write timeout error gone and copied bytes count is 54,347,428. I can see the end-to-end latency reduced to ~20s in P99. I guess the i/o timeout I saw is because the response size too big.
Questions:
In order to get benefits from query split, I need to compress response?
Do you think there are other reasons causing this i/o issue I missed, like http connection setup etc...
Error returned by this io.Copy(w, resp.body) is swallowed and still return success response, why not return error response in case error returned?
May I get some thoughts here?
Environment
I'm running cortex frontend as a docker image in k8s. Nginx configured in k8s receive client requests and forward to cortex frontend. Nginx has timeout set to 60s.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello, I use cortex frontend only in my project, I only enabled query-fronend module with step alignment and splitting in yaml config and I also configured my own downstream prometeus url, below is my config file.
With my load test with k6 to my cortex frontend, I'm seeing most of my requests end-to-end duration is 60s, even though my downstream prometheus returned response in ~14s in P99. After adding tracing and debugging, I find out it's because this io.Copy(w, resp.body) taking time. I printed out error returned by this line, it returned this error: write tcp 10.200.1.15:9090->10.200.0.126:36774: i/o timeout. These requests didn't supply
Accept-Encoding: gzip
in request header.After I supply
Accept-Encoding: gzip
to the request header, the i/o write timeout error gone and copied bytes count is 54,347,428. I can see the end-to-end latency reduced to ~20s in P99. I guess the i/o timeout I saw is because the response size too big.Questions:
In order to get benefits from query split, I need to compress response?
Do you think there are other reasons causing this i/o issue I missed, like http connection setup etc...
Error returned by this io.Copy(w, resp.body) is swallowed and still return success response, why not return error response in case error returned?
May I get some thoughts here?
Environment
I'm running cortex frontend as a docker image in k8s. Nginx configured in k8s receive client requests and forward to cortex frontend. Nginx has timeout set to 60s.
Beta Was this translation helpful? Give feedback.
All reactions