Loadtest: Using pysmb to build a load test on a SMB server. #75
Conversation
|
rebasing on PR #76 to use the pysmb python module instead. |
|
🎉 All dependencies have been resolved ! |
spuiuk
changed the title
spuiuk
force-pushed
the
loadtest
branch
4 times, most recently
from
238ba0a to
e125389
Compare
|
🎉 All dependencies have been resolved ! |
|
Interesting observations:
I am still not convinced that the loading test is picking up errors well. More testing is required. The outputs from the loading test. cephfs ceph-vfs xfs gpfs gpfs-vfs |
|
A second run of the tests
xfs: cephfs: cephfs-vfs gpfs: gpfs-vfs: |
spuiuk
force-pushed
the
loadtest
branch
4 times, most recently
from
2e464c0 to
5a78ab3
Compare
Timeout errors for operations are raised as smb.smb_structs.OperationFailure Not connected errors are raised as base.NotConnectedError We catch them and raise them as more generic errors instead. Signed-off-by: Sachin Prabhu <sp@spui.uk>
These tests load the SMB server and check the results. Signed-off-by: Sachin Prabhu <sp@spui.uk>
This is in preparation for writing larger files over SMB. Signed-off-by: Sachin Prabhu <sp@spui.uk>
Allow us to create a temporary file of a certain size by passing the filesize paramter. Signed-off-by: Sachin Prabhu <sp@spui.uk>
spuiuk
force-pushed
the
loadtest
branch
3 times, most recently
from
0d475da to
0472471
Compare
|
/retest all |
1 similar comment
|
/retest all |
For generating a better small file load, instead of a simple text string, use a 4K files to generate the read/write loads. Also educe number of threads/processs to 50 resulting in 50 x 10 = 500 simultaneous connections. Signed-off-by: Sachin Prabhu <sp@spui.uk>
Signed-off-by: Sachin Prabhu <sp@spui.uk>
|
It appears that ceph.vfs tests do not proceed towards completion. Give below are the test results for xfs, cephfs, gpfs and gpfs-vfs respectively. 10 processes x 5 threads each = 50 simultaneous connection xfs cephfs gpfs gpfs-vfs |
spuiuk
requested review from
anoopcs9,
synarete,
Shwetha-Acharya and
phlogistonjohn
|
The code to test a number of simultaneous connections to the Samba server is nearly complete and is in position to be reviewed. The test with 50 simultaneous connections writing 16Mb files appear to work for most filesystems except for cephfs-vfs. We can further tune the number of connections for cephfs-vfs in later PRs. |
testcases/loading/test_loading.py
Outdated
| @@ -26,17 +26,21 @@ | |||
| # total number of processes | |||
| total_processes: int = 10 | |||
| # each with this number of threads | |||
| per_process_threads: int = 100 | |||
| per_process_threads: int = 50 | |||
There was a problem hiding this comment.
I liked 100 (a round number).
There was a problem hiding this comment.
We have hit the limits of number of files open per process in earlier tests. So to avoid hitting resource limits, I thought it would be better to limit the number of threads per process.
testcases/loading/test_loading.py
Outdated
| # running the connection test for this many seconds | ||
| test_runtime: int = 30 | ||
| # size of test files | ||
| test_file_size = 4 * 1024 # 4k size |
There was a problem hiding this comment.
Python style: 4 * (2 ** 10)
|
/retest centos-ci/cephfs.vfs |
A simple load test
We use python threads to open up several consecutive connections
on the SMB server and perform either open/write, open/read and delete
operations with an interval of 0-0.5 seconds between each operation.
The tests are run for fixed duration of time before we stop and
print out the stats for the number of operations performed
This at the moment is a proof of concept of how we can setup a load test. We are still waiting on #74 to be ready before we can use this change.
The changes proposed are in the last commit. For this code, we set up 100 concurrent connections running for 30 seconds.
depends on #76