[FEATURE] Support Yunikorn as a batch scheduler

[jacobsalway]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

Apache YuniKorn is an alternative Kubernetes scheduler that provides batch scheduling functionality, including gang scheduling, hierarchical queues and job ordering. The Spark operator should support Yunikorn as a batch scheduler by supplying the necessary task group information on the driver pod annotations.

Describe the solution you would like

• Add a Yunikorn scheduler implementation that patches task group annotations and calculates minResources Support gang scheduling with Yunikorn #2107
• Allow a default batch scheduler to be specified so that all applications can be batch scheduled easily Add default batch scheduler argument #2143
• Add a gang scheduling end to end test using Yunikorn to prevent future regressions
  • Prep work Run e2e tests on Kind #2148
• Add a doc page Add page for Spark operator/Yunikorn integration website#3872

Describe alternatives you have considered

Additional context