Testing performance: tricks and tips

[dimakuv]

A useful way of checking CPU utilization: mpstat -P ALL 1 (need to install first)

[dimakuv]

Important note on perf measurement

The numad service is enabled by default on some systems (happened on one Ubuntu 22.04). This service overrides the numactl command. The numad service always kicks in when it sees a "big" process, which our Gramine-VM/TDX process is. And then numad spends a lot of time doing things like this:

sdp@sdp:~$ tail -n 3 /var/log/numad.log
Wed Jul  5 12:58:03 2023: Advising pid 3275237 (qemu) move from nodes (0-1) to nodes (1)
Wed Jul  5 12:58:21 2023: Tried to move PID 3275237, but 460808 pages would not move.
Wed Jul  5 12:58:21 2023: PID 3275237 moved to node(s) 1 in 17.49 seconds

So for performance benchmarking, it's best to disable numad:

systemctl stop numad

Now numactl works, and can be applied like this (to allocate memory only on NUMA node 0 and to execute only on CPUs from NUMA node 0):

numactl --cpunodebind=0 --membind=0  gramine-vm pytorch pytorchexample.py