[BUG] IO hang and never recover

[Wine93]

Background

We use vdbench to test curvefs by specified configure file, and when we increase stress by run fsthrash test, the IO will hang and never recover.

• vdbench

sudo ./vdbench -jn -f profile

# profile
data_errors=1
fsd=fsd1,anchor=/mnt/curvefs/test1/d1,depth=1,width=10,files=50,sizes=(100m,0),shared=yes,openflags=o_direct
fsd=fsd2,anchor=/mnt/curvefs/test2/d2,depth=1,width=10,files=20,sizes=(100m,0),shared=yes,openflags=o_direct
fsd=fsd3,anchor=/mnt/curvefs/test3/d3,depth=1,width=10,files=50,sizes=(300m,0),shared=yes,openflags=o_direct
fwd=fwd1,fsd=fsd*,threads=10,xfersize=(512,20,4k,20,64k,20,512k,20,1024k,20),fileio=random,fileselect=random,rdpct=80
rd=rd1,fwd=fwd*,fwdrate=max,format=restart,elapsed=2000000,interval=1

• fsthrash

./virtualenv/bin/fsthrash-suite --suite_path ./suites --testdir /mnt/curvefs/test2 --numjobs 10

Vdbench Report

Releated Error Logs

I 2023-11-13T10:31:32.104663+0800 692407 client_s3_cache_manager.cpp:501] wait for download object: 1_2129198_374867279_15_0
E 2023-11-13T10:31:32.104766+0800 692406 s3_adapter.cpp:427] GetObject error:  with address : 10.182.30.15 with address : 10.182.30.15
I 2023-11-13T10:31:32.104825+0800 692407 lru_cache.h:749] cached: 1_2129198_374867279_15_0
I 2023-11-13T10:31:32.104832+0800 692407 client_s3_cache_manager.cpp:501] wait for download object: 1_2129198_374867279_15_0
I 2023-11-13T10:31:32.104990+0800 692407 lru_cache.h:749] cached: 1_2129198_374867279_15_0
I 2023-11-13T10:31:32.104996+0800 692407 client_s3_cache_manager.cpp:501] wait for download object: 1_2129198_374867279_15_0
I 2023-11-13T10:31:32.105160+0800 692407 lru_cache.h:749] cached: 1_2129198_374867279_15_0
E 2023-11-13T10:31:32.105156+0800 692403 s3_adapter.cpp:427] GetObject error:  with address : 10.182.30.15 with address : 10.182.30.15
I 2023-11-13T10:31:32.105171+0800 692407 client_s3_cache_manager.cpp:501] wait for download object: 1_2129198_374867279_15_0
E 2023-11-13T10:31:32.105279+0800 692405 s3_adapter.cpp:427] GetObject error:  with address : 10.182.30.15 with address : 10.182.30.15
I 2023-11-13T10:31:32.105329+0800 692407 lru_cache.h:749] cached: 1_2129198_374867279_15_0
I 2023-11-13T10:31:32.105336+0800 692407 client_s3_cache_manager.cpp:501] wait for download object: 1_2129198_374867279_15_0
W 2023-11-13T10:31:32.105333+0800 692404 s3_adapter.cpp:492] HeadObject error:fs2.7.0-bucket1--1_2130973_374890523_8_0--No response body. with address : 10.182.30.15 with address : 10.182.30.15
I 2023-11-13T10:31:32.105343+0800 692404 client_s3.cpp:68] obj 1_2130973_374890523_8_0 seems not exist
E 2023-11-13T10:31:32.105350+0800 692404 client_s3.cpp:71] download error:-2
E 2023-11-13T10:31:32.105358+0800 692404 client_s3_cache_manager.cpp:566] object 1_2130973_374890523_8_0 read from s3 fail, ret = -2
E 2023-11-13T10:31:32.105366+0800 692404 client_s3_cache_manager.cpp:695] read 1_2130973_374890523_8_0 fail
W 2023-11-13T10:31:32.105403+0800 118928 client_s3_cache_manager.cpp:430] read inode = 2130973 retry = 2737, reach max interval = 1000 ms
W 2023-11-13T10:31:32.105494+0800 692407 client_s3_cache_manager.cpp:504] download object: 1_2129198_374867279_15_0 timeout
W 2023-11-13T10:31:32.105846+0800 692403 s3_adapter.cpp:492] HeadObject error:fs2.7.0-bucket1--1_2129198_374867279_15_0--No response body. with address : 10.182.30.15 with address : 10.182.30.15
I 2023-11-13T10:31:32.105859+0800 692403 client_s3.cpp:68] obj 1_2129198_374867279_15_0 seems not exist
E 2023-11-13T10:31:32.105862+0800 692403 client_s3.cpp:71] download error:-2
W 2023-11-13T10:31:32.105862+0800 692406 s3_adapter.cpp:492] HeadObject error:fs2.7.0-bucket1--1_2129198_374867279_15_0--No response body. with address : 10.182.30.15 with address : 10.182.30.15
E 2023-11-13T10:31:32.105871+0800 692403 client_s3_cache_manager.cpp:566] object 1_2129198_374867279_15_0 read from s4 fail, ret = -2
I 2023-11-13T10:31:32.105882+0800 692406 client_s3.cpp:68] obj 1_2129198_374867279_15_0 seems not exist
E 2023-11-13T10:31:32.105887+0800 692406 client_s3.cpp:71] download error:-2
E 2023-11-13T10:31:32.105887+0800 692403 client_s3_cache_manager.cpp:695] read 1_2129198_374867279_15_0 fail
E 2023-11-13T10:31:32.105904+0800 692406 client_s3_cache_manager.cpp:566] object 1_2129198_374867279_15_0 read from s3 fail, ret = -2
E 2023-11-13T10:31:32.105912+0800 692406 client_s3_cache_manager.cpp:695] read 1_2129198_374867279_15_0 fail
W 2023-11-13T10:31:32.105934+0800 118853 client_s3_cache_manager.cpp:430] read inode = 2129198 retry = 2737, reach max interval = 1000 ms
W 2023-11-13T10:31:32.105942+0800 118880 client_s3_cache_manager.cpp:430] read inode = 2129198 retry = 2737, reach max interval = 1000 ms
W 2023-11-13T10:31:32.105980+0800 692405 s3_adapter.cpp:492] HeadObject error:fs2.7.0-bucket1--1_1080198_374842261_5_0--No response body. with address : 10.182.30.15 with address : 10.182.30.15
I 2023-11-13T10:31:32.105994+0800 692405 client_s3.cpp:68] obj 1_1080198_374842261_5_0 seems not exist
E 2023-11-13T10:31:32.105998+0800 692405 client_s3.cpp:71] download error:-2