add GPU accounting for SMHP

[KeitaW]

This PR proposes to add GPU accounting to setup_mariadb_accounting.sh.

$ sreport -tminper cluster utilization --tres="gres/gpu" start=2024-10-21T08:00:00
--------------------------------------------------------------------------------
Cluster Utilization 2024-10-21T08:00:00 - 2024-10-21T08:59:59
Usage reported in TRES Minutes/Percentage of Total
--------------------------------------------------------------------------------
  Cluster      TRES Name         Allocated              Down         PLND Down              Idle           Planned          Reported
--------- -------------- ----------------- ----------------- ----------------- ----------------- ----------------- -----------------
ml-clust+       gres/gpu        11(18.92%)          0(0.00%)          0(0.00%)        49(81.08%)          0(0.00%)       60(100.00%)

[nghtm]

Please test auto-resume with this configuration enabled to confirm there is not a conflict, and post results here. I dont think it should be, since you are only modifying accounting.conf, however I have seen funky behavior with modifying gres attributes, or using gres attributes, in slurm.conf and how hyperpod reacts. Let me know if you require help with testing

[mhuguesaws]

Left comments.

[olegtalalov]

Let me copy paste my comment from slack.

GresTypes=gpu doesn't allow customers to resume a faulty job if they want. AutoResume plugin will always requeue a job in this case. Personally, I'm fine with this because result is the same -> we recover a faulty job from hardware failure. But this is a behaviour change, for existing customers. We can initiate discussion if gres should be on by default, because in my opinion it has more benefits for customers than resuming jobs.

We have CPU only instances. Even trn1/trn2 are marked as "CPU only" and don't have any Gres attached now, and they defenitely will not have gpus. Maybe it make sense to configure these values via ClusterAgent depends on instance types in the cluster.

[KeitaW]

discussing internally on gres setup.