Measuring consumed computational resources

[Nikhil-Desai-Talentica]

Hi Substrate team,

How can I retrieve the information about the computational resources consumed to execute a transaction(both contract call and pallet extrinsic) from a substrate node? I understand that weight is used as a generic measure, but how can I retrieve the weight and how can I figure out how much time was spent doing DB operations, CPU time etc.

[bkchr]

CC @ggwpez

[Nikhil-Desai-Talentica]

thanks for bringing it to Oliver's attention

[ggwpez]

I understand that weight is used as a generic measure, but how can I retrieve the weight and how can I figure out how much time was spent doing DB operations, CPU time etc.

The first dimension of the weight is the time that it took to execute the transaction in pico (10^-12) seconds.
You can already use that as indication. To now compare the time that it took to actually import a block vs its declared weight, you can use benchmark block and pass it --from <BLOCK_NUM> --to <BLOCK_NUM>. This will re-import all those blocks multiple times and measure the effort. It then compares the consumed weight to the declared weight and prints a % number which should always be below 100.
There also exists benchmark extrinsic but it works on an empty DB snapshot, so probably not what you want.
And benchmark storage to measure the average storage access time of a (K, V) pair in a snapshot. Currently it just prints average and a few other metrics, but a histogram plot with Value_size: Key_size would be nice and could help you here.

We dont really have a breakdown chart of how much time a transaction took for DB, CPU. It just benchmarks that all and then outputs one number. The reason for this is that we normally dont want to benchmark with a full state, since it is tiresome to acquire and would often make the benchmarks fail because of invalid values in the snapshot. Otherwise could you describe what you need these numbers for and what you want to archiece?

[Nikhil-Desai-Talentica]

Hi Oliver,

How can I get the weight of executing a transaction on a node using a client written in subxt/substrate-api-client?

[Nikhil-Desai-Talentica]

I want to build a cli tool(substrate client using subxt or substrate-api-client internally) that accepts the pallet name, the extrinsic name and the number of repeats and presents the user with weights estimated by the benchmarking cli and the actual weights.
These are the steps I believe can help me with this

1. Develop some simple pallets
2. Use benchmarking macros to write benchmark tests such that the arguments hit the worst-case time complexity
3. Run benchmark pallet command on each pallet to compute weight estimates (hope to call benchmarking cli tool from my client to do it, so how to do it?)
4. Fill up a block with an extrinsic (want to do it from a client, so how to do it?)
5. Run benchmark block command (again hope to call benchmarking cli tool from my client to do it, so how to do it?)
6. Present the end user with both these metrics (computed estimates and actual ones)

[ggwpez]

Present the end user with both these metrics (computed estimates and actual ones)

You can use the benchmark extrinsic subcommand. It does nearly that; repeat an extrinsic multiple times and print the actual consumed weight.
Currently the supported extrinsics are hard-coded; substrate benchmark extrinsic --dev --list.
Then you can run substrate benchmark extrinsic --dev --pallet "balances" --extrinsic "transfer_keep_alive", it prints:

Executing a balances::transfer_keep_alive extrinsic takes[ns]:
Total: 11168593
Min: 110032, Max: 120137
Average: 111685, Median: 111055, Stddev: 1861.97
Percentiles 99th, 95th, 75th: 118621, 115575, 111933

The CLI can probably be extended to accept any payload from a file, instead of just hard-coded ones. Then you can input that.
Please not that this is not always possible to run any call, since calls depend on specific state. So you would need a way to mock/setup the state prio to building the block. Currently this is not implemented, but could be done via setup extrinsics.

[Nikhil-Desai-Talentica]

Hi Oliver,

first of all, thanks for your time!
I did try that command on the same extrinsic :) How do I extend it to work with extrinsics in custom pallets? By payload, do you mean, the pallet source code?
I am planning on building a unified framework that can benchmark pallets, ink! and solidity smart contracts too and just output the weight
So, I need to call this cli from my tool, and the way things are now, I would have to parse the result of it

[Nikhil-Desai-Talentica]

[Nikhil-Desai-Talentica]

Hi Oliver and Jeff,

Thanks for taking the time to educate me on this. I am considering submitting a proposal towards Implementation benchmarking. I have a few questions about the same. Please help me.

[ggwpez]

So you want to generally compare the performance of ink!, Solang and a pallet?
There was a post in the forum about a similar comparison: Where Polkadot is a Leader in the Blockchain Ecosystem. Maybe you can get some approaches out of it.
The only benchmarks that I wrote so far were all concerned with Pallets and their transactions. Sure you could maybe re-use that for the contracts pallet, but getting an overview first will probably help more.

Generally I think the approach to use the performance as a guide-line for how to build a project a bit weird. ink! vs Pallet are just completely different ways to solve a problem. But okay, not sure which teams you are addressing.