Replies: 5 comments 49 replies
-
|
@fturibak 20GB? ouch. The largest file I was testing was around 3GB. Is the max really 2GB? If it is true then, I guess the mongoDB wont handle it ? You might want to give a v4 a try - it uses only timescaledb (based on postgres) as underlying storage - again I was testing it only with 3GB JTL file. |
Beta Was this translation helpful? Give feedback.
-
|
docker-compose logs db |
Beta Was this translation helpful? Give feedback.
-
|
Finally I've managed to get login screen where I can create user. Looks like the page was cached or something? |
Beta Was this translation helpful? Give feedback.
-
|
that looks correct. Now if you go to |
Beta Was this translation helpful? Give feedback.
-
|
Same problem happens on my enviroment. Initial routing is done to page http://IP_ADRESS:2020/login?returnUrl=%2Fdashboard But if i change it to http://IP_ADRESS:2020/init I can create new user. Can you look into this? Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if I follow here? Could you please describe to me the exact steps how to reproduce it? Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Hello! Can't test upload of large CSV files on v4 cause upload_jtl.py script does not work on v4. I'm getting error: _{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":400,"payload":{"statusCode":400,"error":"Bad Request","message":"Unexpected field"},"headers":{}},"level":"error"} {"level":"info","message":"POST /api/projects/dummyProject/scenarios/dummyScenario/items 400 3862ms","meta":{}}_ Could you advise me please? Tnx |
Beta Was this translation helpful? Give feedback.
-
|
The easiest way is to upload it via UI (although not sure if the nginx does not have a certain limit)? Did you use the Could you share the payload? According to response from the app, you are not sending it correctly. |
Beta Was this translation helpful? Give feedback.
-
|
And yes. via UI upload is of sam kpi file is working .) |
Beta Was this translation helpful? Give feedback.
-
|
Good, and was the huge file processed correctly? I will check the |
Beta Was this translation helpful? Give feedback.
-
|
I fixed the Also note the |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for late response. ~2GB file was process via UI correctly, but it took some time... I'm currently running test which will hopefully upload file larger than 20GB. I've noticed when uploading files around 2GB via Yes, I've done some modifications to
I've done this because I want to have some folder structure where I'll be keeping generated artifacts. i.e. /$BASE_ARTIFACTS_DIR/$APP_VERSION/%Y-%m-%d/$SCENARIO/concurrency_$CONCURRENCY_hold_for_$HOLD_FOR I assume that new version ( v4) does not need error.jtl and that's the reason why existing script was invalid. I'll keep you posted about results |
Beta Was this translation helpful? Give feedback.
-
|
When can I expect v4 release as I can't rely on v3 cause of limits on kpi.jtl file. Thanks |
Beta Was this translation helpful? Give feedback.
-
|
I think you can safely use it if it works for you. I need to still do some testing before final release. |
Beta Was this translation helpful? Give feedback.
-
|
I've successfully uploaded 8GB jtl file to jtl_reporter v4 using upload_jtl.py script. Processing took some time. I didnt measure it, but i would say at least 2 hours. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @SarathKumarEzilarasan, there's https://github.com/ludeknovy/jtl-reporter-listener-service which serves for the same purpose of uploading the data while test is running, but currently there is no JMeter plugin, that supports it. How do you feel about contributing? :) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @ludeknovy , yes would like to do.
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, cool. This problem could be solved by two approaches, either to create JMeter plugin that will send the data to the listener service mentioned above during the test execution, or try to improve the csv parsing/ingestion into the database. Regarding the second solution the related code starts here: https://github.com/ludeknovy/jtl-reporter-be/blob/master/src/server/controllers/item/create-item-controller.ts#L154 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @ludeknovy , Thanks for pointing to the code. I have a question regarding the 1st approach. Cant we call the [jtl-reporter-listener-service] api in the post processor of each request? . Do we really need a Jmeter plugin for this? |
Beta Was this translation helpful? Give feedback.
-
|
In theory, yes, but I don't think it's a good approach. To use the jtl-reporter-listener-service you need to authenticate first, and also start the /async-test-item, only then you can start pushing data to it. Also, it's better to send the data in batches instead of every request individually. Last but not least, it would not be that easy to share this solution with other people. My initial idea for the plugin was to check https://github.com/mderevyankoaqa/jmeter-influxdb2-listener-plugin and see if this could be leveraged and adjusted to fit the usage of this project. |
Beta Was this translation helpful? Give feedback.
-
|
I've noticed some missleading information on graphs Requests statistics reports 0 errors for some Scenario
But when you click on the heart icon, you can see that there was some HTTP500 errors
|
Beta Was this translation helpful? Give feedback.
-
|
I'm thinking that for some test-case scenarios some percentage of error rate would be acceptable, but for some we want to have 0% of error rate because they're crucial and if you have 100 Scenarios it's pretty pain in the ass to click for all of them to the heart icon.. Thanks in advance :-) |
Beta Was this translation helpful? Give feedback.
-
|
would that do it? |
Beta Was this translation helpful? Give feedback.
-
|
Wow, you are fast :-) Sure, this is more than O.K. Don't know if color-blind people would say it's ok... |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
another option. |
Beta Was this translation helpful? Give feedback.
-
Hello!
Is it possible to upload large CSV files ( kpi.jtl ) ?
I'm running some long-running tests (stability tests that runs for several days) that generate *.jtl files larger than 20GB.
Is it possible to upload them to jtl-reporter?
And yes, I've tried modifying internalQueryMaxPushBytes property in mongodb but max size is 2GB.
Thanks
Beta Was this translation helpful? Give feedback.
All reactions