Skip to content

Commit 5ef4652

Browse files
casenavecasenave
committed
add plaid_benchmarks
1 parent 6ce3834 commit 5ef4652

File tree

3 files changed

+191
-1
lines changed

3 files changed

+191
-1
lines changed

‎docs/assets/images/plaid_benchmarks.png‎

34 KB
Loading

‎docs/plaid_benchmarks.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# PLAID Benchmarks
2+
3+
<img src="assets/images/plaid_benchmarks.png" class="align-center" width="60%"
4+
alt="PLAID Benchmarks" />
5+
6+
We provide interactive benchmarks hosted on Hugging Face, in which
7+
anyone can test their own SciML method. These benchmarks involve
8+
regression problems posed on datasets provided in PLAID format. Some of
9+
these datasets have been introduced in the MMGP (Mesh Morphing Gaussian
10+
Process) paper `casenave2023mmgp`, and the PLAID paper
11+
`casenave2025plaid`. A ranking is automatically updated based on a score
12+
computed on the testing set of each dataset. For the benchmarks to be
13+
meaningful, the outputs on the testing sets are not made public.
14+
15+
The relative RMSE is the considered metric for comparing methods. Let
16+
$\{ \mathbf{U}^i_{\rm ref} \}_{i=1}^{n_\star}$ and
17+
$\{ \mathbf{U}^i_{\rm pred} \}_{i=1}^{n_\star}$ be the test observations
18+
and predictions, respectively, of a given field of interest. The
19+
relative RMSE is defined as
20+
21+
$$\mathrm{RRMSE}_f(\mathbf{U}_{\rm ref}, \mathbf{U}_{\rm pred}) = \left( \frac{1}{n_\star}\sum_{i=1}^{n_\star} \frac{\frac{1}{N^i}\|\mathbf{U}^i_{\rm ref} - \mathbf{U}^i_{\rm pred}\|_2^2}{\|\mathbf{U}^i_{\rm ref}\|_\infty^2} \right)^{1/2},$$
22+
23+
where $N^i$ is the number of nodes in the mesh $i$, and
24+
$\max(\mathbf{U}^i_{\rm ref})$ is the maximum entry in the vector
25+
$\mathbf{U}^i_{\rm ref}$. Similarly for scalar outputs:
26+
27+
$$\mathrm{RRMSE}_s(\mathbf{w}_{\rm ref}, \mathbf{w}_{\rm pred}) = \left( \frac{1}{n_\star} \sum_{i=1}^{n_\star} \frac{|w^i_{\rm ref} - w_{\rm pred}^i|^2}{|w^i_{\rm ref}|^2} \right)^{1/2}.$$
28+
29+
## Resources
30+
31+
<table style="width:99%;">
32+
<colgroup>
33+
<col style="width: 19%" />
34+
<col style="width: 48%" />
35+
<col style="width: 31%" />
36+
</colgroup>
37+
<thead>
38+
<tr class="header">
39+
<th></th>
40+
<th><blockquote>
41+
<p>Dataset</p>
42+
</blockquote></th>
43+
<th><blockquote>
44+
<p>Benchmark</p>
45+
</blockquote></th>
46+
</tr>
47+
</thead>
48+
<tbody>
49+
<tr class="odd">
50+
<td><strong>Tensile2d</strong></td>
51+
<td><a
52+
href="https://huggingface.co/datasets/PLAID-datasets/Tensile2d"><img
53+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-md-dark.svg"
54+
alt="Tensile2d_HF" /></a> <a
55+
href="https://doi.org/10.5281/zenodo.14840177"><img
56+
src="https://zenodo.org/badge/DOI/10.5281/zenodo.14840177.svg"
57+
alt="Tensile2d_Z" /></a></td>
58+
<td><a
59+
href="https://huggingface.co/spaces/PLAIDcompetitions/Tensile2dBenchmark"><img
60+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm-dark.svg"
61+
alt="Tensile2d_Be" /></a></td>
62+
</tr>
63+
<tr class="even">
64+
<td><strong>2D_MultiScHypEl</strong></td>
65+
<td><a
66+
href="https://huggingface.co/datasets/PLAID-datasets/2D_Multiscale_Hyperelasticity"><img
67+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-md-dark.svg"
68+
alt="2D_MultiScHypEl_HF" /></a> <a
69+
href="https://doi.org/10.5281/zenodo.14840446"><img
70+
src="https://zenodo.org/badge/DOI/10.5281/zenodo.14840446.svg"
71+
alt="2D_MultiScHypEl_Z" /></a></td>
72+
<td><a
73+
href="https://huggingface.co/spaces/PLAIDcompetitions/2DMultiscaleHyperelasticityBenchmark"><img
74+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm-dark.svg"
75+
alt="2D_MultiScHypEl_Be" /></a></td>
76+
</tr>
77+
<tr class="odd">
78+
<td><strong>2D_ElPlDynamics</strong></td>
79+
<td><a
80+
href="https://huggingface.co/datasets/PLAID-datasets/2D_ElastoPlastoDynamics"><img
81+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-md-dark.svg"
82+
alt="2D_ElPlDynamics_HF" /></a> <a
83+
href="https://doi.org/10.5281/zenodo.15286369"><img
84+
src="https://zenodo.org/badge/DOI/10.5281/zenodo.15286369.svg"
85+
alt="2D_ElPlDynamics_Z" /></a></td>
86+
<td><a
87+
href="https://huggingface.co/spaces/PLAIDcompetitions/2DElastoPlastoDynamics"><img
88+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm-dark.svg"
89+
alt="2D_ElPlDynamics_Be" /></a></td>
90+
</tr>
91+
<tr class="even">
92+
<td><strong>Rotor37</strong></td>
93+
<td><a
94+
href="https://huggingface.co/datasets/PLAID-datasets/Rotor37"><img
95+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-md-dark.svg"
96+
alt="Rotor37_HF" /></a> <a
97+
href="https://doi.org/10.5281/zenodo.14840190"><img
98+
src="https://zenodo.org/badge/DOI/10.5281/zenodo.14840190.svg"
99+
alt="Rotor37_Z" /></a></td>
100+
<td><a
101+
href="https://huggingface.co/spaces/PLAIDcompetitions/Rotor37Benchmark"><img
102+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm-dark.svg"
103+
alt="Rotor37_Be" /></a></td>
104+
</tr>
105+
<tr class="odd">
106+
<td><strong>2D_profile</strong></td>
107+
<td><a
108+
href="https://huggingface.co/datasets/PLAID-datasets/2D_profile"><img
109+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-md-dark.svg"
110+
alt="2D_profile_HF" /></a> <a
111+
href="https://doi.org/10.5281/zenodo.15155119"><img
112+
src="https://zenodo.org/badge/DOI/10.5281/zenodo.15155119.svg"
113+
alt="2D_profile_Z" /></a></td>
114+
<td><a
115+
href="https://huggingface.co/spaces/PLAIDcompetitions/2DprofileBenchmark"><img
116+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm-dark.svg"
117+
alt="2D_profile_Be" /></a></td>
118+
</tr>
119+
<tr class="even">
120+
<td><strong>VKI-LS59</strong></td>
121+
<td><a
122+
href="https://huggingface.co/datasets/PLAID-datasets/VKI-LS59"><img
123+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-md-dark.svg"
124+
alt="VKI-LS59_HF" /></a> <a
125+
href="https://doi.org/10.5281/zenodo.14840512"><img
126+
src="https://zenodo.org/badge/DOI/10.5281/zenodo.14840512.svg"
127+
alt="VKI-LS59_Z" /></a></td>
128+
<td><a
129+
href="https://huggingface.co/spaces/PLAIDcompetitions/VKILS59Benchmark"><img
130+
src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm-dark.svg"
131+
alt="VKI-LS59_Be" /></a></td>
132+
</tr>
133+
</tbody>
134+
</table>
135+
136+
AirfRANS, introduced in `airfrans` is an additional dataset provided in
137+
PLAID format and various variants. Since the outputs on the testing sets
138+
are public, no benchmark application is provided for this dataset.
139+
140+
| | |
141+
|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
142+
| **AirfRANS original** | [![AirfRANS_O_HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-md-dark.svg)](https://huggingface.co/datasets/PLAID-datasets/AirfRANS_original) [![AirfRANS_O_Z](https://zenodo.org/badge/DOI/10.5281/zenodo.14840387.svg)](https://doi.org/10.5281/zenodo.14840387) |
143+
| **AirfRANS clipped** | [![AirfRANS_C_HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-md-dark.svg)](https://huggingface.co/datasets/PLAID-datasets/AirfRANS_clipped) [![AirfRANS_C_Z](https://zenodo.org/badge/DOI/10.5281/zenodo.14840377.svg)](https://doi.org/10.5281/zenodo.14840377) |
144+
| **AirfRANS remeshed** | [![AirfRANS_R_HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/dataset-on-hf-md-dark.svg)](https://huggingface.co/datasets/PLAID-datasets/AirfRANS_remeshed) [![AirfRANS_R_Z](https://zenodo.org/badge/DOI/10.5281/zenodo.14840388.svg)](https://doi.org/10.5281/zenodo.14840388) |
145+
146+
## Benchmark results
147+
148+
As of August 5, 2025
149+
150+
| Dataset | MGN | MMGP | Vi-Transf. | Augur | FNO | MARIO |
151+
|------------------------------------------------|--------|--------|------------|--------|--------|--------|
152+
| <span class="title-ref">Tensile2d</span> | 0.0673 | 0.0026 | 0.0116 | 0.0154 | 0.0123 | 0.0038 |
153+
| <span class="title-ref">2D_MultiScHypEl</span> | 0.0437 | ❌ | 0.0325 | 0.0232 | 0.0302 | 0.0573 |
154+
| <span class="title-ref">2D_ElPlDynamics</span> | 0.1202 | ❌ | 0.0227 | 0.0346 | 0.0215 | 0.0319 |
155+
| <span class="title-ref">Rotor37</span> | 0.0074 | 0.0014 | 0.0029 | 0.0033 | 0.0313 | 0.0017 |
156+
| <span class="title-ref">2D_profile</span> | 0.0593 | 0.0365 | 0.0312 | 0.0425 | 0.0972 | 0.0307 |
157+
| <span class="title-ref">VKI-LS59</span> | 0.0684 | 0.0312 | 0.0193 | 0.0267 | 0.0215 | 0.0124 |
158+
159+
❌: Not compatible with topology variation
160+
161+
<div class="note">
162+
163+
<div class="title">
164+
165+
Note
166+
167+
</div>
168+
169+
- MMGP does not support variable mesh topologies, which limits its
170+
applicability to certain datasets and often necessitates custom
171+
preprocessing for new cases. However, when morphing is either
172+
unnecessary or inexpensive, it offers a highly efficient solution,
173+
combining fast training with good accuracy (e.g., `Tensile2d` and
174+
`Rotor37`).
175+
- MARIO is computationally expensive to train but achieves consistently
176+
a very strong performance across most datasets. Its result on
177+
`2D_MultiScHypEl` is slightly worse than other tested methods, which
178+
may reflect the challenge of capturing complex shape variability in
179+
these cases.
180+
- Vi-Transformer and Augur perform well across all datasets, showing
181+
strong versatility and generalization capabilities.
182+
- FNO suffers on datasets featuring unstructured meshes with pronounced
183+
anisotropies, due to the loss of accuracy introduced by projections to
184+
and from regular grids (e.g., `Rotor37` and `2D_profile`).
185+
Additionally, the use of a 3D regular grid on `Rotor37` results in
186+
substantial computational overhead.
187+
188+
</div>
189+

‎mkdocs.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,5 @@ nav:
9797
- Community:
9898
- Datasets: datasets.md
9999
- Benchmarks: benchmarks.md
100-
- Examples: examples.md
100+
- Examples: examples.md
101+
- Benchmarks: plaid_benchmarks.md

0 commit comments

Comments
 (0)