lotus-bench testing Layer1-X

[scaseye]

when testing the 32GiB sector size can someone explain what is happening in this layering phase?

takes around 15 mins for each layer... no gpu action, only 2 cpu cores...

the 2KiB, and default 512mib take very little time. 512 only has 2 layers at less than a minute a layer, 32gib has over 9 layers... each layer taking ~15min to generate.

makes testing hardware very cumbersome.

2021-05-06T15:28:02.859 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > Layer 1
2021-05-06T15:28:02.859 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > Creating labels for layer 1
2021-05-06T15:28:02.860 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > created label runner
2021-05-06T15:28:02.860 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > created label runner
2021-05-06T15:28:02.860 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > created label runner
2021-05-06T15:43:51.469 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > storing labels on disk
2021-05-06T15:44:09.131 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > generated layer 1 store with id layer-1
2021-05-06T15:44:09.131 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > Layer 2
2021-05-06T15:44:09.131 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > Creating labels for layer 2
2021-05-06T15:44:09.132 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > created label runner
2021-05-06T15:44:09.132 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > created label runner
2021-05-06T15:44:09.132 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > created label runner
2021-05-06T15:59:34.577 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > storing labels on disk
2021-05-06T15:59:55.515 INFO storage_proofs_porep::stacked::vanilla::create_label::multi > generated layer 2 store with id layer-2

[stuberman]

That sounds normal (depending upon hardware and proofs used and configuration).
32GiB sectors require 11 layers averaging 12 minutes or more each.

https://spec.filecoin.io/#section-algorithms.sdr.encoding

[stuberman]

See 11 DRG layers here:

https://spec.filecoin.io/#section-algorithms.sdr.notation.protocol-constants

[stuberman]

The sealing function (DRG layers) provides Proof of Replication (PoRep) that is the basis of Filecoin's decentralized storage design whereby the system (algorithms) assure clients that their data has been successfully stored and then validated every 24 hours through the PoST process. The effort to seal a sector is resource intensive by design to thwart fraud.

https://spec.filecoin.io/#section-algorithms.pos.porep

[scaseye]

Testing on

CPU - 2x AMD EPYC 7502 (32 cores, 2.5GHz each)
GPU - A100 PCIE 40GB

yet during this layer phase the cpu is barely cranking.... no gpu at all.

?

[stuberman]

Correct. PC1 uses one thread, ~60GB ram, some swap and no GPU
GPU is for PC2 and C2