Load testing! #19
Comments
|
Hmm, i just saw an error on http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/irish-uk-border-mapping/warpables.json&scale=30 , the URL became:
|
|
@tech4GT any ideas? |
|
Hmm. http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/lemon-fair-burn/warpables.json&scale=30 is also showing HTTP error 400... :-( |
|
Hmm, are we busting through memory here @icarito ? Where can we pull logs? |
|
I use |
|
Okay, so the first thong about the url transforming is that right now everything is Implemented through GET and hence the redirect with the sequence as query. A rather small refactor will fix that, I'll push that as soon as I can. |
|
@icarito I didn't see a new kubernetes container/cluster, which service did
you set this up in? Thanks!!
…On Mon, Jun 10, 2019, 8:16 PM Varun Gupta ***@***.***> wrote:
Okay, so the first thong about the url transforming is that right now
everything is Implemented through GET and hence the redirect with the
sequence as query. A rather small refactor will fix that, I'll push that as
soon as I can.
For the error, let's see the logs and then I might be able to figure out
why it didn't work!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#19?email_source=notifications&email_token=AAAF6J2P7L7L4KCPBISZETTPZ3VDVA5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXLSHNI#issuecomment-500638645>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAF6J3BY3SVLHEJ6ZQQDZ3PZ3VDVANCNFSM4HWZEIJA>
.
|
|
|
|
Hmm, could be this error? 6/10/19, 5:58 PM |
|
Yeah another log clue |
|
|
@jywarren Could you try running this with v1 API, it'll take considerably more time, but takes far less memory. That should confirm if the problem is being caused by memory constraints. |
|
I also see, near and after that last error, "Finished!Finished!Finished!..." |
|
I will tomorrow! Need to sleep now :-) |
|
Error catching can be done here too! https://console.cloud.google.com/errors/CM_GmvOcrOX8_QE?time=P1D&project=public-lab&authuser=1 |
|
Oh! Good night! |
|
Hmm, @tech4GT when i go to http://34.74.118.242/api/v1/export/?url=http://mapknitter.org/maps/lemon-fair-burn/warpables.json&scale=30, it redirects to: Redirecting is not a problem - we can leave that -- but this second page doesn't work; i get a 400 error. |
|
I'll look into this! Let me try it with a smaller test first! |
|
Okay so the v2 api seems to be working with |
|
Hmm, yes indeed - this looks like it running, but I am still getting a 400... odd! |
|
Okay @jywarren I tried the v1 and that seems to work on the ceres map too!! |
|
Okay we really need to take care of the logs at image sequencer! I'll take care of it this weekend!! So @jywarren I think the issue here might just be with the amount of memory! Since I have not placed a limit to the number of steps that can run concurrently, so it might be that the system just initiates a bunch of steps at the same time but ends up running out of memory to process them later. |
|
Like, lots of these are running. So, I wonder if we are in fact running all the steps, but somehow the original request is receiving a 400? |
|
We can base the limit on the amount of available memory! |
Hmm, this is with the v1, correct? |
|
Great to hear on the ceres map! OK, yeah. We could allow 3 or 5 processes to run concurrently, because in some of these maps we'd be starting 45 processes at once. So this will serialize them a bit? |
|
I'm not sure which actually, i just searched all recent logs on the cluster. It's odd, i can't see the original GET requests themselves. Maybe we need to output to |
|
Okay, I tried forking the process and that does not seem to solve our problem! It still throws the same error. Hmm, maybe it is a problem with a dependency. |
|
@jywarren Is this export correct? The mapknitter map page for https://mapknitter.org/maps/nottingham-city-centre/ takes forever to load. Basically the cloud exporter reboots for some reason when it is under load, this is the result from my local machine! |
|
Update: I am trying out some tweaks and looks like our concurrency limit has a huge effect on the export. Let me figure out what is the max we can do without running into the listeners issue. |
|
Another thing is that our compositing step is linear right now, maybe in version 3 we can make the compositing itself run concurrently! That would make it insanely fast!! |
|
Hmm I raised the concurrency limit to 10 and now it works locally! I even tried it with maps with a lot of images! Like http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/irish-uk-border-mapping/warpables.json&scale=30 |
|
Hmm, i'm not that familiar with concurrency, but is it at all related to
the number of available threads? In a containerized environment maybe there
is some variation of available resources when compared to your local env?
…On Tue, Nov 5, 2019 at 11:18 AM Varun Gupta ***@***.***> wrote:
Hmm I raised the concurrency limit to 10 and now it works locally! I even
tried it with maps with a lot of images! Like
http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/irish-uk-border-mapping/warpables.json&scale=30
But for some reason it's not working in the cloud, the server restarts but
no errors are reported!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#19?email_source=notifications&email_token=AAAF6J3W67CU6FRUIGJFUW3QSGMD7A5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDDIVKY#issuecomment-549882539>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAF6J4BRCYJWFCQRPT3GBDQSGMD7ANCNFSM4HWZEIJA>
.
|
Oh, the concurrency is just the number of sequencer instances we have at any point of time. Still the server restarting in the middle of the export doesn't make sense to me. Well at least it's working locally! I'll do some research as to why this might be happening! |
|
The export shown only looks like 1 image, but the original has many images:
https://mapknitter.org/maps/nottingham-city-centre/
It does seem to be stalling for some reason. I'll check it out, but maybe
try another in the meantime?
…On Tue, Nov 5, 2019 at 12:19 PM Varun Gupta ***@***.***> wrote:
Hmm, i'm not that familiar with concurrency, but is it at all related to
the number of available threads? In a containerized environment maybe there
is some variation of available resources when compared to your local env?
… <#m_7224647282490327134_>
Oh, the concurrency is just the number of sequencer instances we have at
any point of time. Still the server restarting in the middle of the export
doesn't make sense to me. Well at least it's working locally! I'll do some
research as to why this might be happening!
Could you please also confirm if the export I posted above correct?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#19?email_source=notifications&email_token=AAAF6JYO73P3IFNEBDHQHH3QSGTG7A5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDDRCCI#issuecomment-549916937>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAF6JY5OYXAMGV7FBFZPHLQSGTG7ANCNFSM4HWZEIJA>
.
|
|
ok, i got it to load - here:
|
|
Okay tried this on a couple more maps and it seems to work locally! Yay!
Now we just have to figure out why the cloud server reboots! |
|
@jywarren maybe Sebastian would be able to help us with this? The thing is that if the server crashes there are no logs for it. To me it looks like it reboots for no reason! |
|
Hi @jywarren |
|
Of course, no stress at all and good luck!!
…On Thu, Dec 5, 2019, 11:40 PM Varun Gupta ***@***.***> wrote:
Hi @jywarren <https://github.com/jywarren>
Sorry I just saw this one! Actually I am in the middle of university final
evaluations, if I look into this next week would that be okay?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#19?email_source=notifications&email_token=AAAF6J3I3JNCP4EKHUPBRM3QXHJP7A5CNFSM4HWZEIJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGC72MA#issuecomment-562429232>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAF6J3V4XYGI7HSRJK3MCTQXHJP7ANCNFSM4HWZEIJA>
.
|
This comment has been minimized.
This comment has been minimized.
|
I think that's me, sending poorly formatted requests... can be ignored! |
|
So I just tried the problematic URL and while at the same I was looking at the kubernetes pod logs, and realized that the browser response with the status URL isn't being sent until after export process is done. I think it's necessary to end the process and do the processing in a fork. |
|
Okay a clarification from repeating the test:
And blocks for a longer time. Then, when the log starts moving along with: At the same time a response is given to the browser. |
|
Note this line from your log! |
|
Anyways maybe unrelated. |
|
Just noting that I think we are forking here; image-sequencer-app/src/api/v2/index.js Line 65 in ac9b111 Also - @tech4GT is in exams now, but once he's back, i think somehow we aren't seeing the Google Cloud storage version working; I'm receiving back an HTML page even if |
|
Well - i get "still working on it" and then redirected to the HTML page when it's done. I opened this: #36 |
|
Hi @jywarren 😃 |
|
Congratulations!!!!! That's great! You must feel super. We've identified that we need to enable CORS in the app (DONE); see #35 -- and to install SSL utilities to get it running at https (#38). Then the ability to specify an response format in #36. Then we're just trying to debug why this crashes on big exports! Thanks, we're very close!! |
|
Awesome! I'll look into #35 😄 |
|
Moving on to #36 |
|
We are now testing the ruby exporter on production MapKnitter! publiclab/mapknitter#1244 Integration happened here: publiclab/mapknitter#1192 Corresponding implementation with I think if we modify the returned The URL would (once SSL is done in #38) be this instead: https://export-js.mapknitter.org/api/v2/export/?url=http://mapknitter.org/maps/ceres--2/warpables.json&scale=30 |
|
Updated status with in-progress #42 for direct JPG saving (needs help, but started) |
Now, building on the list from publiclab/mapknitter-exporter-sinatra#23, let's see how big images/maps it can handle!
http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/pvdtest/warpables.json&scale=30 is real fast. seconds.
http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/ceres--2/warpables.json&scale=30 -- It worked!!!! 25mb output in <2 minutes! Too big even to upload to GitHub! https://www.dropbox.com/s/m6v131cyynlwx0a/ceres--2.png?dl=0
(note that scale isn't working now -- #14 )
http://34.74.118.242/api/v2/export/?url=http://mapknitter.org/maps/irish-uk-border-mapping/warpables.json&scale=30 next:
Then:
The text was updated successfully, but these errors were encountered: