Provide default headers including user-agent for all provider DAGs #3683
Labels
💻 aspect: code
Concerns the software code in the repository
✨ goal: improvement
Improvement to an existing user-facing feature
🟩 priority: low
Low priority and doesn't need to be rushed
🧱 stack: catalog
Related to the catalog and Airflow DAGs
Problem
Some provider APIs require a User-Agent header, like rawpixel (#3679). In rawpixel's case, we only discovered this when the DAG stopped working.
Description
We could set the User-Agent as a default header on all of our provider APIs by setting it in the
ProviderDataIngester
. Note that currently,headers
is empty on the base class and subclasses totally overwrite it when they want to add headers. We would need to update the base class to provide default headers but still allow subclasses to extend them.The text was updated successfully, but these errors were encountered: