Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Pass extra configuration to json.dump() #622

Closed
wants to merge 2 commits into from

Conversation

Rutam21
Copy link
Contributor

@Rutam21 Rutam21 commented Oct 25, 2024

Description

This PR addresses the need for customizable export configurations when using the await crawler.export_data("export.json") method.

Key Changes:

  • Introduced a new option in the export_data method to accept additional keyword arguments, allowing users to pass configurations directly to json.dump() and other export functions.
  • Created dedicated export helper methods for different formats in the BasicCrawler, including export_data_json and export_data_csv. This separation enhances clarity and maintainability while providing format-specific options.

This enhancement improves the flexibility of data exports, enabling users to tailor their export settings to meet specific requirements.

Issues

Fixes #526

Testing

Unit test cases are added to check the updated configuration and functionality.

Checklist

  • CI passed

@Rutam21
Copy link
Contributor Author

Rutam21 commented Oct 28, 2024

This PR is based on all the discussions in the parent issue and the suggestions thereafter. Please review and suggest changes, if any. Thank you.

cc: @vdusek @janbuchar

Copy link
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure the CI passes, see the CONTRIBUTING.md for more information.

@janbuchar janbuchar self-requested a review October 31, 2024 10:29
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI is still failing, can you please fix the issues?

@Rutam21
Copy link
Contributor Author

Rutam21 commented Oct 31, 2024

@janbuchar I will push a commit to fix some of them by EOD.

@janbuchar
Copy link
Collaborator

@Rutam21 thank you for your initiative, but we decided to just fix up #580 on our own and merge that.

@janbuchar janbuchar closed this Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement/document a way how to pass extra configuration to json.dump()
3 participants