Skip to content

Conversation

@WithoutPants
Copy link
Collaborator

This pull request refactors the scraper package to improve (hopefully) clarity, and prepare for further extensibility.

  • config type (representing the scraper yml file) has been renamed to Definition and config.go has been renamed to definition.go. Definition is a clearer name for it. The sub-sections have had their type names adjusted as well and their definitions tightened up to only include fields related to them.
  • the confusingly named group type has been renamed to definedScraper, to illustrate that it is a scraper derived from a Definition instance. It's constructor has been renamed to scraperFromDefinition, and it's file has been renamed from group.go to defined_scraper.go
  • the mapped.go file has been split into multiple files with each file focused on a particular aspect of the mapping functionality
  • removed reflection from the mapping code that converts the mapped results into the target object. Type-specific functions with concrete types are used instead. This should hopefully resolve any panics related to incorrect type naming. TODO: handle unknown fields in the mapped results

TODO: integration testing

@WithoutPants WithoutPants added this to the Version 0.31.0 milestone Jan 15, 2026
@WithoutPants WithoutPants added the chore Pull requests for refactoring and admin work label Jan 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Pull requests for refactoring and admin work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant