implement an upper bound limit to the number of tracked executor

[ImpSy]

Purpose of this PR

In some case executor can enter in a crashlooping state causing the CR to grow out of control
This can have an impact on the entire SparkApplication CR processing reconciliation loop
This can eventually lead to etcd errors when we trying to patch / update the CR

Proposed changes:

• implement a upper bound limit to the number of tracked executor

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Submitting apps and tracking driver pods is the highest priority of the spark-operator
1 app going haywire should not hinder the entirety of the cluster

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

This feature has been live on our fork for the past year -> spotinst#8

[ImpSy]

Hey @andreyvelich, @jacobsalway, @mwielgus

Could I please have a review on this PR ?

[ImpSy]

@ChenYi015 I've done the change you requested :)
I know that you were not tag on this so thanks for the review 👍

[ImpSy]

@vara-bonthu I've added the possibility to customize the value in the chart following your comment
Thanks for the review 🙏

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ChenYi015]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ChenYi015]

LGTM. Will wait for another approval.

[ImpSy]

@ChenYi015 I case it blocked the merge, I've rebase the PR to fix the conflicts