latex code generation for sampled hyperparameters of diva fbopt

[smilesun]

If we run any benchmark configuration file, we get a hyperparameters.csv file where each line represent a hyperparameter configuration for a particular algorithm, now we want to show that in a paper, which we need to format into latex. We need a script to transform the csv file into latex for display in a paper.

Below is an example of the benchmark folder where one could find hyperparameters.csv

├── commit.txt
├── config.txt
├── graphics
│ ├── diva_fbopt_full
│ ├── radar_dist.png
│ ├── radar.png
│ ├── scatterpl
│ ├── sp_matrix_dist.png
│ ├── sp_matrix_dist_reg.png
│ ├── sp_matrix.png
│ ├── sp_matrix_reg.png
│ └── variational_plots
├── hyperparameters.csv
├── results.csv
└── rule_results
├── 0.csv
├── 1.csv
├── 2.csv
├── 3.csv
├── 4.csv
├── 5.csv
├── 6.csv
└── 7.csv

In the results plot, we have

We need to let the reader know what each index correspond to which hyperparameter using the hyperparameter.csv:

cat zoutput/benchmarks/pacs_benchmark_resnet50_model_only/hyperparameter
s.csv
,method,model,params
0,erm,erm,{}
1,diva,diva,"{'gamma_y': 6173000, 'gamma_d': 17000, 'zx_dim': 64, 'zy_dim': 64, 'zd_dim': 32}"
2,diva,diva,"{'gamma_y': 563000, 'gamma_d': 36906000, 'zx_dim': 96, 'zy_dim': 64, 'zd_dim': 96}"
3,diva,diva,"{'gamma_y': 1871000, 'gamma_d': 50395000, 'zx_dim': 0, 'zy_dim': 32, 'zd_dim': 32}"
4,diva,diva,"{'gamma_y': 1568000, 'gamma_d': 7257000, 'zx_dim': 64, 'zy_dim': 64, 'zd_dim': 64}"
5,diva,diva,"{'gamma_y': 114000, 'gamma_d': 12501000, 'zx_dim': 32, 'zy_dim': 64, 'zd_dim': 32}"
6,diva,diva,"{'gamma_y': 2808000, 'gamma_d': 2936000, 'zx_dim': 96, 'zy_dim': 64, 'zd_dim': 64}"
7,diva,diva,"{'gamma_y': 183000, 'gamma_d': 285000, 'zx_dim': 64, 'zy_dim': 64, 'zd_dim': 96}"
8,diva,diva,"{'gamma_y': 12961000, 'gamma_d': 30203000, 'zx_dim': 96, 'zy_dim': 96, 'zd_dim': 64}"
9,hduva,hduva,"{'gamma_y': 183000, 'zy_dim': 64, 'zd_dim': 96}"
10,hduva,hduva,"{'gamma_y': 2808000, 'zy_dim': 64, 'zd_dim': 64}"
11,hduva,hduva,"{'gamma_y': 6173000, 'zy_dim': 64, 'zd_dim': 32}"
12,hduva,hduva,"{'gamma_y': 563000, 'zy_dim': 64, 'zd_dim': 96}"
13,hduva,hduva,"{'gamma_y': 1568000, 'zy_dim': 64, 'zd_dim': 64}"
14,hduva,hduva,"{'gamma_y': 114000, 'zy_dim': 64, 'zd_dim': 32}"
15,hduva,hduva,"{'gamma_y': 1871000, 'zy_dim': 32, 'zd_dim': 32}"
16,hduva,hduva,"{'gamma_y': 12961000, 'zy_dim': 96, 'zd_dim': 64}"
17,dann,dann,{'gamma_reg': 0.46}
18,dann,dann,{'gamma_reg': 0.03}
19,dann,dann,{'gamma_reg': 0.56}
20,dann,dann,{'gamma_reg': 21.400000000000002}
21,dann,dann,{'gamma_reg': 1.3}
22,dann,dann,{'gamma_reg': 0.030000000000000002}
23,dann,dann,{'gamma_reg': 2.9499999999999997}
24,dann,dann,{'gamma_reg': 3.83}

[smilesun]

in fbopt branch,

./run_benchmark_standalone https://github.com/marrlab/DomainLab/blob/fbopt/examples/benchmark/pacs_diva_fbopt_and_baselines.yaml

or ./run_benchmark_slurm https://github.com/marrlab/DomainLab/blob/fbopt/examples/benchmark/pacs_diva_fbopt_and_baselines.yaml

snakemake will first sample the hyperparameters, before other jobs being runed, so hyperparameters.csv will be generated, we need a way to put that into latex.

[e-dorigatti]

code

import pandas as pd
import json

df = pd.read_csv(f)
df = df.merge(
    pd.json_normalize(df['params'].apply(lambda x: json.loads(x.replace("'", '"')))),
    left_index=True, right_index=True
).rename(columns={
    'Unnamed: 0': 'Configuration',
    'method': 'Method', 'model': 'Model',
    #'k_i_gain_ratio': 
    'mu_clip': '$\mu_{clip}$',
    'gamma_d': '$\gamma_d$',
    'gamma_y': '$\gamma_y$',
    'gamma_reg': '$\gamma_{reg}$',
    'zx_dim': 'dim($z_x$)',
    'zy_dim': 'dim($z_y$)',
    'zd_dim': 'dim($z_d$)',
}).drop(columns=['params'])
print(df.to_latex(na_rep='--', index=False, escape=False))

result with the csv above:

\begin{tabular}{rllrrrrrr}
\toprule
 Configuration & Method & Model &  $\gamma_y$ &  $\gamma_d$ &  dim($z_x$) &  dim($z_y$) &  dim($z_d$) &  $\gamma_{reg}$ \\
\midrule
             0 &    erm &   erm &          -- &          -- &          -- &          -- &          -- &              -- \\
             1 &   diva &  diva &   6173000.0 &     17000.0 &        64.0 &        64.0 &        32.0 &              -- \\
             2 &   diva &  diva &    563000.0 &  36906000.0 &        96.0 &        64.0 &        96.0 &              -- \\
             3 &   diva &  diva &   1871000.0 &  50395000.0 &         0.0 &        32.0 &        32.0 &              -- \\
             4 &   diva &  diva &   1568000.0 &   7257000.0 &        64.0 &        64.0 &        64.0 &              -- \\
             5 &   diva &  diva &    114000.0 &  12501000.0 &        32.0 &        64.0 &        32.0 &              -- \\
             6 &   diva &  diva &   2808000.0 &   2936000.0 &        96.0 &        64.0 &        64.0 &              -- \\
             7 &   diva &  diva &    183000.0 &    285000.0 &        64.0 &        64.0 &        96.0 &              -- \\
             8 &   diva &  diva &  12961000.0 &  30203000.0 &        96.0 &        96.0 &        64.0 &              -- \\
             9 &  hduva & hduva &    183000.0 &          -- &          -- &        64.0 &        96.0 &              -- \\
            10 &  hduva & hduva &   2808000.0 &          -- &          -- &        64.0 &        64.0 &              -- \\
            11 &  hduva & hduva &   6173000.0 &          -- &          -- &        64.0 &        32.0 &              -- \\
            12 &  hduva & hduva &    563000.0 &          -- &          -- &        64.0 &        96.0 &              -- \\
            13 &  hduva & hduva &   1568000.0 &          -- &          -- &        64.0 &        64.0 &              -- \\
            14 &  hduva & hduva &    114000.0 &          -- &          -- &        64.0 &        32.0 &              -- \\
            15 &  hduva & hduva &   1871000.0 &          -- &          -- &        32.0 &        32.0 &              -- \\
            16 &  hduva & hduva &  12961000.0 &          -- &          -- &        96.0 &        64.0 &              -- \\
            17 &   dann &  dann &          -- &          -- &          -- &          -- &          -- &            0.46 \\
            18 &   dann &  dann &          -- &          -- &          -- &          -- &          -- &            0.03 \\
            19 &   dann &  dann &          -- &          -- &          -- &          -- &          -- &            0.56 \\
            20 &   dann &  dann &          -- &          -- &          -- &          -- &          -- &           21.40 \\
            21 &   dann &  dann &          -- &          -- &          -- &          -- &          -- &            1.30 \\
            22 &   dann &  dann &          -- &          -- &          -- &          -- &          -- &            0.03 \\
            23 &   dann &  dann &          -- &          -- &          -- &          -- &          -- &            2.95 \\
            24 &   dann &  dann &          -- &          -- &          -- &          -- &          -- &            3.83 \\
\bottomrule
\end{tabular}