GH-273 add benchmarks

[hmottestad]

Add benchmarks.

[hmottestad]

@filip26 I assume you're on holiday :) Norway has holidays in July, but the rest of Europe typically has in August. Just wanted to let you know that this PR is the first step in my plan for merging in the performance optimizations. Each PR will build on the previous, so I won't start preparing the next one until this one is merged. That way we can measure the impact of each PR.

[hmottestad]

Any chance you've had a chance to look at this PR @filip26 ?

[filip26]

Hi @hmottestad ,
I'm sorry for the long wait, I will try to find time this weekend.

[hmottestad]

@filip26 have you had a chance to look this over yet?

[filip26]

@hmottestad for some reason I wasn't able commit to the PR directly, here is a branch resolving all the conflicts, please check it and update the PR. Thank you.

I'm thinking about using Github actions and virtual machines to run the benchmarks. What do you think? Is there an action we can use to run it? Or would you be willing to write it? or do you have other thoughts?

https://github.com/filip26/titanium-json-ld/tree/GH-273-add-benchmark

[hmottestad]

I usually prefer comments on the PR that I can resolve. I can't really compare your branch with mine, could you explain what you've changed?

[filip26]

@hmottestad sorry, as you can see the PR has a conflict with main branch, pom_parent.xml has changed. I've only resolved the conflict, that's all.

[hmottestad]

Ahh, ok, I can fix that by rebasing onto main.

My experience with running the benchmarks is simply to run them locally on my laptop on the main branch and then on the branch that I'm working on to compare the performance. I've not heard of anyone running JMH benchmarks with Github Actions. I know the JMH can output the benchmark results as JSON, so you could try to write a custom Action to run it before and after and then parse the json and create some kind of comparison.

One general issue with using Github Actions is that there is no way to control what hardware you run on or who else is running at the same time.

[filip26]

The issue with running benchmarks locally is that the results cannot be compared, as there is no reference machine configuration. Also, it complicates an approval, as a maintainer has to run it locally too, possibly getting different results.

I have no deep knowledge of GitHub infrastructure, even I would assume they use the same configuration across (public) repositories. Which is not good assertion, but better than using local machines where configuration will be totally different (not counting OS, other applications running in the background, etc.)

Well, if nobody uses it, perhaps, there is an opportunity for some to get a credit, and maybe $, for leading ;)

[hmottestad]

I've usually trusted the contributors when they show a performance gain for a code change.

I'll leave making a GitHub Action to you.

[filip26]

Thanks for the generosity, but benchmarks are not my concern and my plate is full already ;)

[hmottestad]

My understanding is that you want Github Actions and that you don't want to merge this PR without it. So I've gone ahead and closed this PR since neither of us wants to implement that. Please let me know if I've misunderstood.