[Proving Issue] wdpost failed.

[c02pine]

Describe the problem

A wdpost error has occurred on the 1 deadline since 5 days ago.
Deadlines 0,3,4 are normal except for deadline 1.
https://filecoinproject.slack.com/archives/CPFTWMY7N/p1621748919105000

Version

lotus version 1.9.0+mainnet+git.ada7f97ba.dirty

Setup

CPU : Ryzen threadripper 3970x
RAM : 256 GB
GPU : GeForce RTX 3090

Run Daemon : service lotus-daemon start
Run Miner : service lotus-miner start

Miner Env :

Enviromnet=LOTUS_PATH=/root/.lotus
Environment=FIL_PROOFS_PARAMETER_CACHE=/seal_1/FIL_Parameter
Environment=FIL_PROOFS_PARENT_CACHE=/lotus/FIL_Parent
Environment=TMPDIR=/lotus/tmp
# Performence
Environment=RUSTFLAGS="-C target-cpu=native -g"
Environment=FFI_BUILD_FROM_SOURCE=1
#Environment=FIL_PROOFS_USE_MULTICORE_SDR=1
#Environment=BELLMAN_CPU_UTILIZATION=1
Environment=FIL_PROOFS_MAXIMIZE_CACHING=1
Environment=FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
Environment=FIL_PROOFS_USE_GPU_TREE_BUILDER=1
Environment=FIL_PROOFS_SDR_PARENTS_CACHE_SIZE=1073741824
Environment=TRUST_PARAMS=1
#Environment=BELLMAN_CUSTOM_GPU="GeForce RTX 3090:10496"
Environment=CUDA_VISIBLE_DEVICES=0
# Log
Environment=GOLOG_FILE="/lotus/log/miner.log"
Environment=GOLOG_LOG_FMT="json"
Environment=RUST_LOG=info
Environment=RUST_BACKTRACE=1

LimitNOFILE=1048576:1048576

Proving status

Miner: f0435045
Current Epoch:           785565
Proving Period Boundary: 2656
Proving Period Start:    783136 (20 hours 14 minutes ago)
Next Period Start:       786016 (in 3 hours 45 minutes)

Faults:      1622 (39.60%)
Recovering:  0
Deadline Index:       40
Deadline Sectors:     0
Deadline Open:        785536 (14 minutes 30 seconds ago)
Deadline Close:       785596 (in 15 minutes 30 seconds)
Deadline Challenge:   785516 (24 minutes 30 seconds ago)
Deadline FaultCutoff: 785466 (49 minutes 30 seconds ago)

deadline  partitions  sectors (faults)  proven partitions
0         1           2349 (0)          0
1         1           1623 (1622)       0
2         1           90 (0)            0
3         1           37 (0)            0
4         0           0 (0)             0
5         0           0 (0)             0
6         0           0 (0)             0
7         0           0 (0)             0
8         0           0 (0)             0
9         0           0 (0)             0
10        0           0 (0)             0
11        0           0 (0)             0
12        0           0 (0)             0
13        0           0 (0)             0
14        0           0 (0)             0
15        0           0 (0)             0
16        0           0 (0)             0
17        0           0 (0)             0
18        0           0 (0)             0
19        0           0 (0)             0
20        0           0 (0)             0
21        0           0 (0)             0
22        0           0 (0)             0
23        0           0 (0)             0
24        0           0 (0)             0
25        0           0 (0)             0
26        0           0 (0)             0
27        0           0 (0)             0
28        0           0 (0)             0
29        0           0 (0)             0
30        0           0 (0)             0
31        0           0 (0)             0
32        0           0 (0)             0
33        0           0 (0)             0
34        0           0 (0)             0
35        0           0 (0)             0
36        0           0 (0)             0
37        0           0 (0)             0
38        0           0 (0)             0
39        0           0 (0)             0
40        0           0 (0)             0  (current)
41        0           0 (0)             0
42        0           0 (0)             0
43        0           0 (0)             0
44        0           0 (0)             0
45        0           0 (0)             0
46        0           0 (0)             0
47        0           0 (0)             0

Lotus miner logs

{"level":"warn","ts":"2021-05-23T05:35:02.626+0900","logger":"storageminer","caller":"storage/wdpost_changehandler.go:252","msg":"Aborted window post Proving (Deadline: &{CurrentEpoch:780306 PeriodStart:780256 Index:1 Open:780316 Close:780376 Challenge:780296 FaultCutoff:780246 WPoStPeriodDeadlines:48 WPoStProvingPeriod:2880 WPoStChallengeWindow:60 WPoStChallengeLookback:20 FaultDeclarationCutoff:70})"}
{"level":"warn","ts":"2021-05-23T04:33:06.124+0900","logger":"storageminer","caller":"storage/wdpost_run.go:314","msg":"declare faults recovered Message CID","cid":"bafy2bzaced73dn226gv42hoxedlocpzeybm4da3bxf73ndcauejvoq2u6fdga"}
{"level":"warn","ts":"2021-05-23T05:33:05.134+0900","logger":"storageminer","caller":"storage/wdpost_run.go:226","msg":"Checked sectors","checked":1568,"good":1568}
{"level":"info","ts":"2021-05-23T05:33:05.180+0900","logger":"storageminer","caller":"storage/wdpost_run.go:584","msg":"running window post","chain-random":"GTrAvQGWo3rViO4/hGMXxZHfQTICsnapzheO3F8Gk+U=","deadline":{"CurrentEpoch":780306,"PeriodStart":780256,"Index":1,"Open":780316,"Close":780376,"Challenge":780296,"FaultCutoff":780246,"WPoStPeriodDeadlines":48,"WPoStProvingPeriod":2880,"WPoStChallengeWindow":60,"WPoStChallengeLookback":20,"FaultDeclarationCutoff":70},"height":"780306","skipped":0}
{"level":"info","ts":"2021-05-23T05:35:02.626+0900","logger":"storageminer","caller":"storage/wdpost_run.go:600","msg":"computing window post","batch":0,"elapsed":117.445467858}
{"level":"error","ts":"2021-05-23T05:35:02.626+0900","logger":"storageminer","caller":"storage/wdpost_run.go:104","msg":"runPost failed: running window post failed:
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPost
/root/lotus/storage/wdpost_run.go:650
- generate_window_post: merkle_tree failed: SectorId(3706)
Caused by:
Resource temporarily unavailable (os error 11)
Stack backtrace:
0: <merkletree::store::level_cache::LevelCacheStore<E,R> as merkletree::store::Store<E>>::new_with_config
1: storage_proofs_core::merkle::builders::create_tree
2: filecoin_proofs::types::private_replica_info::PrivateReplicaInfo<Tree>::merkle_tree
3: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold\
4: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
5: core::iter::adapters::process_results
6: filecoin_proofs::api::window_post::generate_window_post
7: filecoin_proofs_api::post::generate_window_post_inner
8: filecoin_proofs_api::post::generate_window_post
9: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
10: ffi_toolkit::catch_panic_response
11: fil_generate_window_post
12: _cgo_a8fa62747d41_Cfunc_fil_generate_window_post
at ./tmp/go-build/cgo-gcc-prolog:596:11
13: runtime.asmcgocall
at ./usr/local/go/src/runtime/asm_amd64.s:656
github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt
/root/lotus/extern/filecoin-ffi/proofs.go:587
github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt
/root/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:45
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPost
/root/lotus/storage/wdpost_run.go:597
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runGeneratePoST
/root/lotus/storage/wdpost_run.go:102
github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).startGeneratePoST.func1\n    
\t/root/lotus/storage/wdpost_run.go:86
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1374"}

[f8-ptrk]

run

lotus-miner proving check 0

and

lotus-miner proving check --slow 0

what does it say for sector 3706?

[maweis1981]

How do you fixed this problem, I saw your sectors all normal yet now.

[maweis1981]

fil@fil-postminer-01:/disk_md0/logs$ lotus-miner proving check 0
deadline partition sector status
0 0 5 good
0 0 7 good
0 0 2 good
0 0 3 good
0 0 4 good
fil@fil-postminer-01:/disk_md0/logs$ lotus-miner proving check 01
deadline partition sector status
1 0 6 good

[funktafide]

Did you find a solution to this issue? I'm having the same problem. All proving checks are good. Still cannot past post.