Apache Spark Receiver

[Caleb-Hurshman]

The purpose and use-cases of the new component

The purpose of this component is to allow monitoring of Apache Spark applications and jobs through the collection of performance metrics like memory utilization, CPU utilization, executor duration, etc. gathered through the exposed Spark REST API.

Example configuration for the component

The configuration parameters are all optional:

• endpoint (default: localhost:4040)

receivers:
  spark:
    endpoint: localhost:4040

Telemetry data types supported

Metrics

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

No sponsor yet

Additional context

No response

[atoulme]

There is some ongoing work with the databricks receiver that may intersect with this proposal. @pmcollins fyi

[djaglowski]

There is some ongoing work with the databricks receiver that may intersect with this proposal.

@atoulme, can you clarify what you're referring to here?

I'm willing to sponsor this unless there is some reason it would be unified with another component.

[atoulme]

See this issue for context: #18724

[djaglowski]

Thanks @atoulme. I agree there's some overlap.

I think work can proceed on this one, though we should look for ways to make some portion of this receiver's codebase reuseable by the databricks receiver.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.