Seaweedfs cycling on self hosted #4151
Description
Self-Hosted Version
26.1.0
CPU Architecture
x86_64
Docker Version
29.1.3
Docker Compose Version
5.0.0
Machine Specification
- My system meets the minimum system requirements of Sentry
Installation Type
Upgrade from 25.12
Steps to Reproduce
Just started up, and sentry-self-hosted-seaweedfs-1 flapping:
docker logs -f sentry-self-hosted-seaweedfs-1
I0201 11:37:16.742496 volume_loading.go:139 checking volume data integrity for volume 3575
I0201 11:37:16.742522 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3575.ldb
I0201 11:37:16.742893 disk_location.go:191 data file /data/nodestore_3560.dat, replication=000 v=3 size=15095472 ttl=3M disk_id=0
I0201 11:37:16.742904 disk_location.go:191 data file /data/nodestore_3559.dat, replication=000 v=3 size=15284720 ttl=3M disk_id=0
I0201 11:37:16.743010 volume_loading.go:98 readSuperBlock volume 3576 version 3
I0201 11:37:16.743016 volume_loading.go:98 readSuperBlock volume 3577 version 3
I0201 11:37:16.743049 volume_loading.go:139 checking volume data integrity for volume 3576
I0201 11:37:16.743063 volume_loading.go:139 checking volume data integrity for volume 3577
I0201 11:37:16.743085 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3576.ldb
I0201 11:37:16.743089 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3577.ldb
I0201 11:37:16.752354 volume_loading.go:204 loading leveldb /data/idx/nodestore_3570.ldb error: open /data/idx/nodestore_3570.ldb/MANIFEST-004530: too many open files
I0201 11:37:16.752626 volume_loading.go:204 loading leveldb /data/idx/nodestore_3563.ldb error: open /data/idx/nodestore_3563.ldb: too many open files
I0201 11:37:16.791831 disk_location.go:183 new volume nodestore_3570 error open /data/idx/nodestore_3570.ldb/MANIFEST-004530: too many open files
I0201 11:37:16.791867 disk_location.go:183 new volume nodestore_3563 error open /data/idx/nodestore_3563.ldb: too many open files
I0201 11:37:16.791978 volume_loading.go:98 readSuperBlock volume 3579 version 3
I0201 11:37:16.791984 volume_loading.go:98 readSuperBlock volume 3578 version 3
I0201 11:37:16.792006 volume_loading.go:139 checking volume data integrity for volume 3579
I0201 11:37:16.792023 volume_loading.go:139 checking volume data integrity for volume 3578
I0201 11:37:16.792024 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3579.ldb
I0201 11:37:16.792053 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3578.ldb
I0201 11:37:16.792084 volume_loading.go:204 loading leveldb /data/idx/nodestore_3578.ldb error: open /data/idx/nodestore_3578.ldb/LOCK: too many open files
I0201 11:37:16.792089 volume_loading.go:204 loading leveldb /data/idx/nodestore_3579.ldb error: open /data/idx/nodestore_3579.ldb/LOCK: too many open files
I0201 11:37:16.797936 disk_location.go:183 new volume nodestore_3579 error open /data/idx/nodestore_3579.ldb/LOCK: too many open files
I0201 11:37:16.797949 disk_location.go:183 new volume nodestore_3578 error open /data/idx/nodestore_3578.ldb/LOCK: too many open files
I0201 11:37:16.798040 volume_loading.go:98 readSuperBlock volume 3580 version 3
I0201 11:37:16.798068 volume_loading.go:139 checking volume data integrity for volume 3580
I0201 11:37:16.798067 volume_loading.go:98 readSuperBlock volume 3581 version 3
I0201 11:37:16.798082 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3580.ldb
I0201 11:37:16.798102 volume_loading.go:139 checking volume data integrity for volume 3581
I0201 11:37:16.798133 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3581.ldb
I0201 11:37:16.798176 volume_loading.go:204 loading leveldb /data/idx/nodestore_3581.ldb error: open /data/idx/nodestore_3581.ldb/LOCK: too many open files
I0201 11:37:16.798264 volume_loading.go:204 loading leveldb /data/idx/nodestore_3574.ldb error: open /data/idx/nodestore_3574.ldb/MANIFEST-004518: too many open files
I0201 11:37:16.798319 disk_location.go:183 new volume nodestore_3581 error open /data/idx/nodestore_3581.ldb/LOCK: too many open files
I0201 11:37:16.798403 disk_location.go:183 new volume nodestore_3574 error open /data/idx/nodestore_3574.ldb/MANIFEST-004518: too many open files
I0201 11:37:16.798410 volume_loading.go:98 readSuperBlock volume 3582 version 3
I0201 11:37:16.798432 volume_loading.go:139 checking volume data integrity for volume 3582
I0201 11:37:16.798446 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3582.ldb
I0201 11:37:16.798495 volume_loading.go:98 readSuperBlock volume 3583 version 3
I0201 11:37:16.798564 volume_loading.go:139 checking volume data integrity for volume 3583
I0201 11:37:16.798563 volume_loading.go:204 loading leveldb /data/idx/nodestore_3575.ldb error: open /data/idx/nodestore_3575.ldb/MANIFEST-004509: too many open files
I0201 11:37:16.798588 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3583.ldb
I0201 11:37:16.798693 volume_loading.go:204 loading leveldb /data/idx/nodestore_3576.ldb error: open /data/idx/nodestore_3576.ldb/CURRENT: too many open files
I0201 11:37:16.812896 disk_location.go:183 new volume nodestore_3575 error open /data/idx/nodestore_3575.ldb/MANIFEST-004509: too many open files
I0201 11:37:16.812971 disk_location.go:183 new volume nodestore_3576 error open /data/idx/nodestore_3576.ldb/CURRENT: too many open files
I0201 11:37:16.813029 volume_loading.go:98 readSuperBlock volume 3584 version 3
I0201 11:37:16.813062 volume_loading.go:139 checking volume data integrity for volume 3584
I0201 11:37:16.813073 volume_loading.go:98 readSuperBlock volume 3585 version 3
I0201 11:37:16.813093 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3584.ldb
I0201 11:37:16.813096 volume_loading.go:139 checking volume data integrity for volume 3585
I0201 11:37:16.813122 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3585.ldb
I0201 11:37:16.813154 volume_loading.go:204 loading leveldb /data/idx/nodestore_3585.ldb error: open /data/idx/nodestore_3585.ldb/LOCK: too many open files
I0201 11:37:16.813240 volume_loading.go:204 loading leveldb /data/idx/nodestore_3584.ldb error: open /data/idx/nodestore_3584.ldb: too many open files
I0201 11:37:16.826514 disk_location.go:183 new volume nodestore_3585 error open /data/idx/nodestore_3585.ldb/LOCK: too many open files
I0201 11:37:16.826520 disk_location.go:183 new volume nodestore_3584 error open /data/idx/nodestore_3584.ldb: too many open files
I0201 11:37:16.826637 volume_loading.go:98 readSuperBlock volume 3587 version 3
I0201 11:37:16.826657 volume_loading.go:98 readSuperBlock volume 3586 version 3
I0201 11:37:16.826666 volume_loading.go:139 checking volume data integrity for volume 3587
I0201 11:37:16.826687 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3587.ldb
I0201 11:37:16.826704 volume_loading.go:139 checking volume data integrity for volume 3586
I0201 11:37:16.826728 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3586.ldb
I0201 11:37:16.826735 volume_loading.go:204 loading leveldb /data/idx/nodestore_3587.ldb error: open /data/idx/nodestore_3587.ldb/LOG: too many open files
I0201 11:37:16.826802 volume_loading.go:204 loading leveldb /data/idx/nodestore_3586.ldb error: open /data/idx/nodestore_3586.ldb/LOG: too many open files
I0201 11:37:16.831344 disk_location.go:183 new volume nodestore_3587 error open /data/idx/nodestore_3587.ldb/LOG: too many open files
I0201 11:37:16.831416 disk_location.go:183 new volume nodestore_3586 error open /data/idx/nodestore_3586.ldb/LOG: too many open files
I0201 11:37:16.831445 volume_loading.go:98 readSuperBlock volume 3588 version 3
I0201 11:37:16.831466 volume_loading.go:139 checking volume data integrity for volume 3588
I0201 11:37:16.831487 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3588.ldb
I0201 11:37:16.831542 volume_loading.go:98 readSuperBlock volume 3589 version 3
I0201 11:37:16.831578 volume_loading.go:139 checking volume data integrity for volume 3589
I0201 11:37:16.831600 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3589.ldb
I0201 11:37:16.831660 volume_loading.go:204 loading leveldb /data/idx/nodestore_3589.ldb error: open /data/idx/nodestore_3589.ldb/LOG: too many open files
I0201 11:37:16.832016 needle_map_leveldb.go:66 Loading /data/idx/nodestore_3564.ldb... , watermark: 0
I0201 11:37:16.832015 volume_loading.go:204 loading leveldb /data/idx/nodestore_3568.ldb error: open /data/idx/nodestore_3568.ldb: too many open files
I0201 11:37:16.832239 volume_loading.go:204 loading leveldb /data/idx/nodestore_3582.ldb error: open /data/idx/nodestore_3582.ldb/CURRENT: too many open files
I0201 11:37:16.832252 volume_loading.go:204 loading leveldb /data/idx/nodestore_3583.ldb error: open /data/idx/nodestore_3583.ldb/CURRENT: too many open files
I0201 11:37:16.833223 disk_location.go:191 data file /data/nodestore_3564.dat, replication=000 v=3 size=15464088 ttl=3M disk_id=0
I0201 11:37:16.833307 volume_loading.go:98 readSuperBlock volume 359 version 3
I0201 11:37:16.833332 volume_loading.go:139 checking volume data integrity for volume 359
I0201 11:37:16.833359 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_359.ldb
I0201 11:37:16.842039 disk_location.go:183 new volume nodestore_3568 error open /data/idx/nodestore_3568.ldb: too many open files
I0201 11:37:16.842059 disk_location.go:183 new volume nodestore_3583 error open /data/idx/nodestore_3583.ldb/CURRENT: too many open files
I0201 11:37:16.842101 disk_location.go:183 new volume nodestore_3582 error open /data/idx/nodestore_3582.ldb/CURRENT: too many open files
I0201 11:37:16.842156 volume_loading.go:98 readSuperBlock volume 3590 version 3
I0201 11:37:16.842185 volume_loading.go:139 checking volume data integrity for volume 3590
I0201 11:37:16.842159 volume_loading.go:98 readSuperBlock volume 3591 version 3
I0201 11:37:16.842210 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3590.ldb
I0201 11:37:16.842231 volume_loading.go:98 readSuperBlock volume 3592 version 3
I0201 11:37:16.842063 disk_location.go:183 new volume nodestore_3589 error open /data/idx/nodestore_3589.ldb/LOG: too many open files
I0201 11:37:16.842273 volume_loading.go:139 checking volume data integrity for volume 3592
I0201 11:37:16.842300 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3592.ldb
I0201 11:37:16.842332 volume_loading.go:98 readSuperBlock volume 3593 version 3
I0201 11:37:16.842345 volume_loading.go:204 loading leveldb /data/idx/nodestore_3590.ldb error: open /data/idx/nodestore_3590.ldb: too many open files
I0201 11:37:16.842359 volume_loading.go:139 checking volume data integrity for volume 3593
I0201 11:37:16.842236 volume_loading.go:139 checking volume data integrity for volume 3591
I0201 11:37:16.842414 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3591.ldb
I0201 11:37:16.842378 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3593.ldb
I0201 11:37:16.842481 volume_loading.go:204 loading leveldb /data/idx/nodestore_3593.ldb error: open /data/idx/nodestore_3593.ldb/LOCK: too many open files
I0201 11:37:16.842451 volume_loading.go:204 loading leveldb /data/idx/nodestore_3591.ldb error: open /data/idx/nodestore_3591.ldb/LOCK: too many open files
I0201 11:37:16.842809 volume_loading.go:204 loading leveldb /data/idx/nodestore_3592.ldb error: open /data/idx/nodestore_3592.ldb: too many open files
I0201 11:37:16.848098 disk_location.go:183 new volume nodestore_3590 error open /data/idx/nodestore_3590.ldb: too many open files
I0201 11:37:16.848131 disk_location.go:183 new volume nodestore_3591 error open /data/idx/nodestore_3591.ldb/LOCK: too many open files
I0201 11:37:16.848130 disk_location.go:183 new volume nodestore_3592 error open /data/idx/nodestore_3592.ldb: too many open files
I0201 11:37:16.848099 disk_location.go:183 new volume nodestore_3593 error open /data/idx/nodestore_3593.ldb/LOCK: too many open files
I0201 11:37:16.848231 volume_loading.go:98 readSuperBlock volume 3595 version 3
I0201 11:37:16.848197 volume_loading.go:98 readSuperBlock volume 3594 version 3
I0201 11:37:16.848268 volume_loading.go:139 checking volume data integrity for volume 3595
I0201 11:37:16.848277 volume_loading.go:98 readSuperBlock volume 3596 version 3
I0201 11:37:16.848288 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3595.ldb
I0201 11:37:16.848304 volume_loading.go:139 checking volume data integrity for volume 3596
I0201 11:37:16.848302 volume_loading.go:139 checking volume data integrity for volume 3594
I0201 11:37:16.848323 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3596.ldb
I0201 11:37:16.848344 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3594.ldb
I0201 11:37:16.848376 volume_loading.go:98 readSuperBlock volume 3597 version 3
I0201 11:37:16.848435 volume_loading.go:204 loading leveldb /data/idx/nodestore_3596.ldb error: open /data/idx/nodestore_3596.ldb: too many open files
I0201 11:37:16.848485 disk_location.go:183 new volume nodestore_3597 error cannot write Volume Index /data/idx/nodestore_3597.idx: open /data/idx/nodestore_3597.idx: too many open files
I0201 11:37:16.848542 disk_location.go:183 new volume nodestore_3596 error open /data/idx/nodestore_3596.ldb: too many open files
I0201 11:37:16.848583 volume_loading.go:98 readSuperBlock volume 3598 version 3
I0201 11:37:16.848637 volume_loading.go:139 checking volume data integrity for volume 3598
I0201 11:37:16.848657 volume_loading.go:98 readSuperBlock volume 3599 version 3
I0201 11:37:16.848667 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3598.ldb
I0201 11:37:16.848679 volume_loading.go:139 checking volume data integrity for volume 3599
I0201 11:37:16.848742 volume_loading.go:204 loading leveldb /data/idx/nodestore_3594.ldb error: open /data/idx/nodestore_3594.ldb/004495.log: too many open files
I0201 11:37:16.848761 volume_loading.go:204 loading leveldb /data/idx/nodestore_3566.ldb error: open /data/idx/nodestore_3566.ldb: too many open files
I0201 11:37:16.848771 volume_loading.go:204 loading leveldb /data/idx/nodestore_3565.ldb error: open /data/idx/nodestore_3565.ldb: too many open files
I0201 11:37:16.848777 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3599.ldb
I0201 11:37:16.861927 disk_location.go:183 new volume nodestore_3594 error open /data/idx/nodestore_3594.ldb/004495.log: too many open files
I0201 11:37:16.861981 disk_location.go:183 new volume nodestore_3565 error open /data/idx/nodestore_3565.ldb: too many open files
I0201 11:37:16.861973 disk_location.go:183 new volume nodestore_3566 error open /data/idx/nodestore_3566.ldb: too many open files
I0201 11:37:16.862068 volume_loading.go:98 readSuperBlock volume 3600 version 3
I0201 11:37:16.862096 volume_loading.go:98 readSuperBlock volume 3602 version 3
I0201 11:37:16.862102 volume_loading.go:98 readSuperBlock volume 3601 version 3
I0201 11:37:16.862120 volume_loading.go:139 checking volume data integrity for volume 3602
I0201 11:37:16.862127 volume_loading.go:139 checking volume data integrity for volume 3601
I0201 11:37:16.862144 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3602.ldb
I0201 11:37:16.862144 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3601.ldb
I0201 11:37:16.862103 volume_loading.go:139 checking volume data integrity for volume 3600
I0201 11:37:16.862224 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3600.ldb
I0201 11:37:16.862294 volume_loading.go:204 loading leveldb /data/idx/nodestore_3600.ldb error: open /data/idx/nodestore_3600.ldb/LOG: too many open files
I0201 11:37:16.862501 volume_loading.go:204 loading leveldb /data/idx/nodestore_3567.ldb error: open /data/idx/nodestore_3567.ldb: too many open files
I0201 11:37:16.862581 volume_loading.go:204 loading leveldb /data/idx/nodestore_3577.ldb error: open /data/idx/nodestore_3577.ldb: too many open files
I0201 11:37:16.871042 disk_location.go:183 new volume nodestore_3600 error open /data/idx/nodestore_3600.ldb/LOG: too many open files
I0201 11:37:16.871087 disk_location.go:183 new volume nodestore_3567 error open /data/idx/nodestore_3567.ldb: too many open files
I0201 11:37:16.871102 disk_location.go:183 new volume nodestore_3577 error open /data/idx/nodestore_3577.ldb: too many open files
I0201 11:37:16.871148 volume_loading.go:98 readSuperBlock volume 3603 version 3
I0201 11:37:16.871192 volume_loading.go:98 readSuperBlock volume 3604 version 3
I0201 11:37:16.871196 volume_loading.go:139 checking volume data integrity for volume 3603
I0201 11:37:16.871213 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3603.ldb
I0201 11:37:16.871217 volume_loading.go:98 readSuperBlock volume 3605 version 3
I0201 11:37:16.871226 volume_loading.go:139 checking volume data integrity for volume 3604
I0201 11:37:16.871245 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3604.ldb
I0201 11:37:16.871262 volume_loading.go:139 checking volume data integrity for volume 3605
I0201 11:37:16.871290 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3605.ldb
I0201 11:37:16.871299 volume_loading.go:204 loading leveldb /data/idx/nodestore_3604.ldb error: open /data/idx/nodestore_3604.ldb/LOG: too many open files
I0201 11:37:16.871338 volume_loading.go:204 loading leveldb /data/idx/nodestore_3603.ldb error: open /data/idx/nodestore_3603.ldb: too many open files
I0201 11:37:16.873004 needle_map_leveldb.go:66 Loading /data/idx/nodestore_357.ldb... , watermark: 0
I0201 11:37:16.873032 needle_map_leveldb.go:66 Loading /data/idx/nodestore_3569.ldb... , watermark: 0
I0201 11:37:16.873071 needle_map_leveldb.go:66 Loading /data/idx/nodestore_3571.ldb... , watermark: 0
I0201 11:37:16.873121 volume_loading.go:204 loading leveldb /data/idx/nodestore_3599.ldb error: open /data/idx/nodestore_3599.ldb/CURRENT: too many open files
I0201 11:37:16.873216 needle_map_leveldb.go:66 Loading /data/idx/nodestore_3572.ldb... , watermark: 0
I0201 11:37:16.873268 volume_loading.go:204 loading leveldb /data/idx/nodestore_3595.ldb error: open /data/idx/nodestore_3595.ldb/CURRENT: too many open files
I0201 11:37:16.874320 disk_location.go:191 data file /data/nodestore_3569.dat, replication=000 v=3 size=15160768 ttl=3M disk_id=0
I0201 11:37:16.874361 disk_location.go:191 data file /data/nodestore_3571.dat, replication=000 v=3 size=15129216 ttl=3M disk_id=0
I0201 11:37:16.874408 disk_location.go:191 data file /data/nodestore_357.dat, replication=000 v=3 size=17848776 ttl=3M disk_id=0
I0201 11:37:16.874418 disk_location.go:191 data file /data/nodestore_3572.dat, replication=000 v=3 size=15271896 ttl=3M disk_id=0
I0201 11:37:16.874448 volume_loading.go:98 readSuperBlock volume 3606 version 3
W0201 11:37:16.874461 volume_info.go:39 fail to read /data/nodestore_3609.vif : open /data/nodestore_3609.vif: too many open files
I0201 11:37:16.874491 volume_loading.go:139 checking volume data integrity for volume 3606
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdb5ecc]
goroutine 342 [running]:
os.(*File).Name(...)
/usr/local/go/src/os/file.go:62
github.com/seaweedfs/seaweedfs/weed/storage/backend.NewDiskFile(0x0)
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/backend/disk_file.go:28 +0x4c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xca38f5c160, 0x1, 0x1, 0x3, 0x0, 0x3)
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/volume_loading.go:76 +0x3d1
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7fff6f533da6, 0x5}, {0x7fff6f533e94, 0x9}, {0xc011667e90, 0x9}, 0xe17, 0x3, 0x0, 0x0, ...)
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/volume.go:66 +0x1b7
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc00076d680, {0x409fc28, 0xc01166e580}, 0x3, 0x1, 0x0, 0x0)
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/disk_location.go:181 +0x390
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/disk_location.go:222 +0xb1
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes in goroutine 315
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/disk_location.go:219 +0xd3
Expected Result
Smooth start
Actual Result
docker logs -f sentry-self-hosted-seaweedfs-1
0201 11:37:16.742496 volume_loading.go:139 checking volume data integrity for volume 3575
I0201 11:37:16.742522 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3575.ldb
I0201 11:37:16.742893 disk_location.go:191 data file /data/nodestore_3560.dat, replication=000 v=3 size=15095472 ttl=3M disk_id=0
I0201 11:37:16.742904 disk_location.go:191 data file /data/nodestore_3559.dat, replication=000 v=3 size=15284720 ttl=3M disk_id=0
I0201 11:37:16.743010 volume_loading.go:98 readSuperBlock volume 3576 version 3
I0201 11:37:16.743016 volume_loading.go:98 readSuperBlock volume 3577 version 3
I0201 11:37:16.743049 volume_loading.go:139 checking volume data integrity for volume 3576
I0201 11:37:16.743063 volume_loading.go:139 checking volume data integrity for volume 3577
I0201 11:37:16.743085 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3576.ldb
I0201 11:37:16.743089 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3577.ldb
I0201 11:37:16.752354 volume_loading.go:204 loading leveldb /data/idx/nodestore_3570.ldb error: open /data/idx/nodestore_3570.ldb/MANIFEST-004530: too many open files
I0201 11:37:16.752626 volume_loading.go:204 loading leveldb /data/idx/nodestore_3563.ldb error: open /data/idx/nodestore_3563.ldb: too many open files
I0201 11:37:16.791831 disk_location.go:183 new volume nodestore_3570 error open /data/idx/nodestore_3570.ldb/MANIFEST-004530: too many open files
I0201 11:37:16.791867 disk_location.go:183 new volume nodestore_3563 error open /data/idx/nodestore_3563.ldb: too many open files
I0201 11:37:16.791978 volume_loading.go:98 readSuperBlock volume 3579 version 3
I0201 11:37:16.791984 volume_loading.go:98 readSuperBlock volume 3578 version 3
I0201 11:37:16.792006 volume_loading.go:139 checking volume data integrity for volume 3579
I0201 11:37:16.792023 volume_loading.go:139 checking volume data integrity for volume 3578
I0201 11:37:16.792024 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3579.ldb
I0201 11:37:16.792053 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3578.ldb
I0201 11:37:16.792084 volume_loading.go:204 loading leveldb /data/idx/nodestore_3578.ldb error: open /data/idx/nodestore_3578.ldb/LOCK: too many open files
I0201 11:37:16.792089 volume_loading.go:204 loading leveldb /data/idx/nodestore_3579.ldb error: open /data/idx/nodestore_3579.ldb/LOCK: too many open files
I0201 11:37:16.797936 disk_location.go:183 new volume nodestore_3579 error open /data/idx/nodestore_3579.ldb/LOCK: too many open files
I0201 11:37:16.797949 disk_location.go:183 new volume nodestore_3578 error open /data/idx/nodestore_3578.ldb/LOCK: too many open files
I0201 11:37:16.798040 volume_loading.go:98 readSuperBlock volume 3580 version 3
I0201 11:37:16.798068 volume_loading.go:139 checking volume data integrity for volume 3580
I0201 11:37:16.798067 volume_loading.go:98 readSuperBlock volume 3581 version 3
I0201 11:37:16.798082 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3580.ldb
I0201 11:37:16.798102 volume_loading.go:139 checking volume data integrity for volume 3581
I0201 11:37:16.798133 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3581.ldb
I0201 11:37:16.798176 volume_loading.go:204 loading leveldb /data/idx/nodestore_3581.ldb error: open /data/idx/nodestore_3581.ldb/LOCK: too many open files
I0201 11:37:16.798264 volume_loading.go:204 loading leveldb /data/idx/nodestore_3574.ldb error: open /data/idx/nodestore_3574.ldb/MANIFEST-004518: too many open files
I0201 11:37:16.798319 disk_location.go:183 new volume nodestore_3581 error open /data/idx/nodestore_3581.ldb/LOCK: too many open files
I0201 11:37:16.798403 disk_location.go:183 new volume nodestore_3574 error open /data/idx/nodestore_3574.ldb/MANIFEST-004518: too many open files
I0201 11:37:16.798410 volume_loading.go:98 readSuperBlock volume 3582 version 3
I0201 11:37:16.798432 volume_loading.go:139 checking volume data integrity for volume 3582
I0201 11:37:16.798446 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3582.ldb
I0201 11:37:16.798495 volume_loading.go:98 readSuperBlock volume 3583 version 3
I0201 11:37:16.798564 volume_loading.go:139 checking volume data integrity for volume 3583
I0201 11:37:16.798563 volume_loading.go:204 loading leveldb /data/idx/nodestore_3575.ldb error: open /data/idx/nodestore_3575.ldb/MANIFEST-004509: too many open files
I0201 11:37:16.798588 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3583.ldb
I0201 11:37:16.798693 volume_loading.go:204 loading leveldb /data/idx/nodestore_3576.ldb error: open /data/idx/nodestore_3576.ldb/CURRENT: too many open files
I0201 11:37:16.812896 disk_location.go:183 new volume nodestore_3575 error open /data/idx/nodestore_3575.ldb/MANIFEST-004509: too many open files
I0201 11:37:16.812971 disk_location.go:183 new volume nodestore_3576 error open /data/idx/nodestore_3576.ldb/CURRENT: too many open files
I0201 11:37:16.813029 volume_loading.go:98 readSuperBlock volume 3584 version 3
I0201 11:37:16.813062 volume_loading.go:139 checking volume data integrity for volume 3584
I0201 11:37:16.813073 volume_loading.go:98 readSuperBlock volume 3585 version 3
I0201 11:37:16.813093 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3584.ldb
I0201 11:37:16.813096 volume_loading.go:139 checking volume data integrity for volume 3585
I0201 11:37:16.813122 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3585.ldb
I0201 11:37:16.813154 volume_loading.go:204 loading leveldb /data/idx/nodestore_3585.ldb error: open /data/idx/nodestore_3585.ldb/LOCK: too many open files
I0201 11:37:16.813240 volume_loading.go:204 loading leveldb /data/idx/nodestore_3584.ldb error: open /data/idx/nodestore_3584.ldb: too many open files
I0201 11:37:16.826514 disk_location.go:183 new volume nodestore_3585 error open /data/idx/nodestore_3585.ldb/LOCK: too many open files
I0201 11:37:16.826520 disk_location.go:183 new volume nodestore_3584 error open /data/idx/nodestore_3584.ldb: too many open files
I0201 11:37:16.826637 volume_loading.go:98 readSuperBlock volume 3587 version 3
I0201 11:37:16.826657 volume_loading.go:98 readSuperBlock volume 3586 version 3
I0201 11:37:16.826666 volume_loading.go:139 checking volume data integrity for volume 3587
I0201 11:37:16.826687 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3587.ldb
I0201 11:37:16.826704 volume_loading.go:139 checking volume data integrity for volume 3586
I0201 11:37:16.826728 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3586.ldb
I0201 11:37:16.826735 volume_loading.go:204 loading leveldb /data/idx/nodestore_3587.ldb error: open /data/idx/nodestore_3587.ldb/LOG: too many open files
I0201 11:37:16.826802 volume_loading.go:204 loading leveldb /data/idx/nodestore_3586.ldb error: open /data/idx/nodestore_3586.ldb/LOG: too many open files
I0201 11:37:16.831344 disk_location.go:183 new volume nodestore_3587 error open /data/idx/nodestore_3587.ldb/LOG: too many open files
I0201 11:37:16.831416 disk_location.go:183 new volume nodestore_3586 error open /data/idx/nodestore_3586.ldb/LOG: too many open files
I0201 11:37:16.831445 volume_loading.go:98 readSuperBlock volume 3588 version 3
I0201 11:37:16.831466 volume_loading.go:139 checking volume data integrity for volume 3588
I0201 11:37:16.831487 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3588.ldb
I0201 11:37:16.831542 volume_loading.go:98 readSuperBlock volume 3589 version 3
I0201 11:37:16.831578 volume_loading.go:139 checking volume data integrity for volume 3589
I0201 11:37:16.831600 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3589.ldb
I0201 11:37:16.831660 volume_loading.go:204 loading leveldb /data/idx/nodestore_3589.ldb error: open /data/idx/nodestore_3589.ldb/LOG: too many open files
I0201 11:37:16.832016 needle_map_leveldb.go:66 Loading /data/idx/nodestore_3564.ldb... , watermark: 0
I0201 11:37:16.832015 volume_loading.go:204 loading leveldb /data/idx/nodestore_3568.ldb error: open /data/idx/nodestore_3568.ldb: too many open files
I0201 11:37:16.832239 volume_loading.go:204 loading leveldb /data/idx/nodestore_3582.ldb error: open /data/idx/nodestore_3582.ldb/CURRENT: too many open files
I0201 11:37:16.832252 volume_loading.go:204 loading leveldb /data/idx/nodestore_3583.ldb error: open /data/idx/nodestore_3583.ldb/CURRENT: too many open files
I0201 11:37:16.833223 disk_location.go:191 data file /data/nodestore_3564.dat, replication=000 v=3 size=15464088 ttl=3M disk_id=0
I0201 11:37:16.833307 volume_loading.go:98 readSuperBlock volume 359 version 3
I0201 11:37:16.833332 volume_loading.go:139 checking volume data integrity for volume 359
I0201 11:37:16.833359 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_359.ldb
I0201 11:37:16.842039 disk_location.go:183 new volume nodestore_3568 error open /data/idx/nodestore_3568.ldb: too many open files
I0201 11:37:16.842059 disk_location.go:183 new volume nodestore_3583 error open /data/idx/nodestore_3583.ldb/CURRENT: too many open files
I0201 11:37:16.842101 disk_location.go:183 new volume nodestore_3582 error open /data/idx/nodestore_3582.ldb/CURRENT: too many open files
I0201 11:37:16.842156 volume_loading.go:98 readSuperBlock volume 3590 version 3
I0201 11:37:16.842185 volume_loading.go:139 checking volume data integrity for volume 3590
I0201 11:37:16.842159 volume_loading.go:98 readSuperBlock volume 3591 version 3
I0201 11:37:16.842210 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3590.ldb
I0201 11:37:16.842231 volume_loading.go:98 readSuperBlock volume 3592 version 3
I0201 11:37:16.842063 disk_location.go:183 new volume nodestore_3589 error open /data/idx/nodestore_3589.ldb/LOG: too many open files
I0201 11:37:16.842273 volume_loading.go:139 checking volume data integrity for volume 3592
I0201 11:37:16.842300 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3592.ldb
I0201 11:37:16.842332 volume_loading.go:98 readSuperBlock volume 3593 version 3
I0201 11:37:16.842345 volume_loading.go:204 loading leveldb /data/idx/nodestore_3590.ldb error: open /data/idx/nodestore_3590.ldb: too many open files
I0201 11:37:16.842359 volume_loading.go:139 checking volume data integrity for volume 3593
I0201 11:37:16.842236 volume_loading.go:139 checking volume data integrity for volume 3591
I0201 11:37:16.842414 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3591.ldb
I0201 11:37:16.842378 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3593.ldb
I0201 11:37:16.842481 volume_loading.go:204 loading leveldb /data/idx/nodestore_3593.ldb error: open /data/idx/nodestore_3593.ldb/LOCK: too many open files
I0201 11:37:16.842451 volume_loading.go:204 loading leveldb /data/idx/nodestore_3591.ldb error: open /data/idx/nodestore_3591.ldb/LOCK: too many open files
I0201 11:37:16.842809 volume_loading.go:204 loading leveldb /data/idx/nodestore_3592.ldb error: open /data/idx/nodestore_3592.ldb: too many open files
I0201 11:37:16.848098 disk_location.go:183 new volume nodestore_3590 error open /data/idx/nodestore_3590.ldb: too many open files
I0201 11:37:16.848131 disk_location.go:183 new volume nodestore_3591 error open /data/idx/nodestore_3591.ldb/LOCK: too many open files
I0201 11:37:16.848130 disk_location.go:183 new volume nodestore_3592 error open /data/idx/nodestore_3592.ldb: too many open files
I0201 11:37:16.848099 disk_location.go:183 new volume nodestore_3593 error open /data/idx/nodestore_3593.ldb/LOCK: too many open files
I0201 11:37:16.848231 volume_loading.go:98 readSuperBlock volume 3595 version 3
I0201 11:37:16.848197 volume_loading.go:98 readSuperBlock volume 3594 version 3
I0201 11:37:16.848268 volume_loading.go:139 checking volume data integrity for volume 3595
I0201 11:37:16.848277 volume_loading.go:98 readSuperBlock volume 3596 version 3
I0201 11:37:16.848288 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3595.ldb
I0201 11:37:16.848304 volume_loading.go:139 checking volume data integrity for volume 3596
I0201 11:37:16.848302 volume_loading.go:139 checking volume data integrity for volume 3594
I0201 11:37:16.848323 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3596.ldb
I0201 11:37:16.848344 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3594.ldb
I0201 11:37:16.848376 volume_loading.go:98 readSuperBlock volume 3597 version 3
I0201 11:37:16.848435 volume_loading.go:204 loading leveldb /data/idx/nodestore_3596.ldb error: open /data/idx/nodestore_3596.ldb: too many open files
I0201 11:37:16.848485 disk_location.go:183 new volume nodestore_3597 error cannot write Volume Index /data/idx/nodestore_3597.idx: open /data/idx/nodestore_3597.idx: too many open files
I0201 11:37:16.848542 disk_location.go:183 new volume nodestore_3596 error open /data/idx/nodestore_3596.ldb: too many open files
I0201 11:37:16.848583 volume_loading.go:98 readSuperBlock volume 3598 version 3
I0201 11:37:16.848637 volume_loading.go:139 checking volume data integrity for volume 3598
I0201 11:37:16.848657 volume_loading.go:98 readSuperBlock volume 3599 version 3
I0201 11:37:16.848667 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3598.ldb
I0201 11:37:16.848679 volume_loading.go:139 checking volume data integrity for volume 3599
I0201 11:37:16.848742 volume_loading.go:204 loading leveldb /data/idx/nodestore_3594.ldb error: open /data/idx/nodestore_3594.ldb/004495.log: too many open files
I0201 11:37:16.848761 volume_loading.go:204 loading leveldb /data/idx/nodestore_3566.ldb error: open /data/idx/nodestore_3566.ldb: too many open files
I0201 11:37:16.848771 volume_loading.go:204 loading leveldb /data/idx/nodestore_3565.ldb error: open /data/idx/nodestore_3565.ldb: too many open files
I0201 11:37:16.848777 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3599.ldb
I0201 11:37:16.861927 disk_location.go:183 new volume nodestore_3594 error open /data/idx/nodestore_3594.ldb/004495.log: too many open files
I0201 11:37:16.861981 disk_location.go:183 new volume nodestore_3565 error open /data/idx/nodestore_3565.ldb: too many open files
I0201 11:37:16.861973 disk_location.go:183 new volume nodestore_3566 error open /data/idx/nodestore_3566.ldb: too many open files
I0201 11:37:16.862068 volume_loading.go:98 readSuperBlock volume 3600 version 3
I0201 11:37:16.862096 volume_loading.go:98 readSuperBlock volume 3602 version 3
I0201 11:37:16.862102 volume_loading.go:98 readSuperBlock volume 3601 version 3
I0201 11:37:16.862120 volume_loading.go:139 checking volume data integrity for volume 3602
I0201 11:37:16.862127 volume_loading.go:139 checking volume data integrity for volume 3601
I0201 11:37:16.862144 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3602.ldb
I0201 11:37:16.862144 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3601.ldb
I0201 11:37:16.862103 volume_loading.go:139 checking volume data integrity for volume 3600
I0201 11:37:16.862224 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3600.ldb
I0201 11:37:16.862294 volume_loading.go:204 loading leveldb /data/idx/nodestore_3600.ldb error: open /data/idx/nodestore_3600.ldb/LOG: too many open files
I0201 11:37:16.862501 volume_loading.go:204 loading leveldb /data/idx/nodestore_3567.ldb error: open /data/idx/nodestore_3567.ldb: too many open files
I0201 11:37:16.862581 volume_loading.go:204 loading leveldb /data/idx/nodestore_3577.ldb error: open /data/idx/nodestore_3577.ldb: too many open files
I0201 11:37:16.871042 disk_location.go:183 new volume nodestore_3600 error open /data/idx/nodestore_3600.ldb/LOG: too many open files
I0201 11:37:16.871087 disk_location.go:183 new volume nodestore_3567 error open /data/idx/nodestore_3567.ldb: too many open files
I0201 11:37:16.871102 disk_location.go:183 new volume nodestore_3577 error open /data/idx/nodestore_3577.ldb: too many open files
I0201 11:37:16.871148 volume_loading.go:98 readSuperBlock volume 3603 version 3
I0201 11:37:16.871192 volume_loading.go:98 readSuperBlock volume 3604 version 3
I0201 11:37:16.871196 volume_loading.go:139 checking volume data integrity for volume 3603
I0201 11:37:16.871213 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3603.ldb
I0201 11:37:16.871217 volume_loading.go:98 readSuperBlock volume 3605 version 3
I0201 11:37:16.871226 volume_loading.go:139 checking volume data integrity for volume 3604
I0201 11:37:16.871245 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3604.ldb
I0201 11:37:16.871262 volume_loading.go:139 checking volume data integrity for volume 3605
I0201 11:37:16.871290 volume_loading.go:202 loading leveldb large index /data/idx/nodestore_3605.ldb
I0201 11:37:16.871299 volume_loading.go:204 loading leveldb /data/idx/nodestore_3604.ldb error: open /data/idx/nodestore_3604.ldb/LOG: too many open files
I0201 11:37:16.871338 volume_loading.go:204 loading leveldb /data/idx/nodestore_3603.ldb error: open /data/idx/nodestore_3603.ldb: too many open files
I0201 11:37:16.873004 needle_map_leveldb.go:66 Loading /data/idx/nodestore_357.ldb... , watermark: 0
I0201 11:37:16.873032 needle_map_leveldb.go:66 Loading /data/idx/nodestore_3569.ldb... , watermark: 0
I0201 11:37:16.873071 needle_map_leveldb.go:66 Loading /data/idx/nodestore_3571.ldb... , watermark: 0
I0201 11:37:16.873121 volume_loading.go:204 loading leveldb /data/idx/nodestore_3599.ldb error: open /data/idx/nodestore_3599.ldb/CURRENT: too many open files
I0201 11:37:16.873216 needle_map_leveldb.go:66 Loading /data/idx/nodestore_3572.ldb... , watermark: 0
I0201 11:37:16.873268 volume_loading.go:204 loading leveldb /data/idx/nodestore_3595.ldb error: open /data/idx/nodestore_3595.ldb/CURRENT: too many open files
I0201 11:37:16.874320 disk_location.go:191 data file /data/nodestore_3569.dat, replication=000 v=3 size=15160768 ttl=3M disk_id=0
I0201 11:37:16.874361 disk_location.go:191 data file /data/nodestore_3571.dat, replication=000 v=3 size=15129216 ttl=3M disk_id=0
I0201 11:37:16.874408 disk_location.go:191 data file /data/nodestore_357.dat, replication=000 v=3 size=17848776 ttl=3M disk_id=0
I0201 11:37:16.874418 disk_location.go:191 data file /data/nodestore_3572.dat, replication=000 v=3 size=15271896 ttl=3M disk_id=0
I0201 11:37:16.874448 volume_loading.go:98 readSuperBlock volume 3606 version 3
W0201 11:37:16.874461 volume_info.go:39 fail to read /data/nodestore_3609.vif : open /data/nodestore_3609.vif: too many open files
I0201 11:37:16.874491 volume_loading.go:139 checking volume data integrity for volume 3606
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdb5ecc]
goroutine 342 [running]:
os.(*File).Name(...)
/usr/local/go/src/os/file.go:62
github.com/seaweedfs/seaweedfs/weed/storage/backend.NewDiskFile(0x0)
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/backend/disk_file.go:28 +0x4c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xca38f5c160, 0x1, 0x1, 0x3, 0x0, 0x3)
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/volume_loading.go:76 +0x3d1
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7fff6f533da6, 0x5}, {0x7fff6f533e94, 0x9}, {0xc011667e90, 0x9}, 0xe17, 0x3, 0x0, 0x0, ...)
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/volume.go:66 +0x1b7
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc00076d680, {0x409fc28, 0xc01166e580}, 0x3, 0x1, 0x0, 0x0)
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/disk_location.go:181 +0x390
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/disk_location.go:222 +0xb1
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes in goroutine 315
/go/src/github.com/seaweedfs/seaweedfs/weed/storage/disk_location.go:219 +0xd3
Event ID
No response