Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iterator that supports multiple items within files #56

Open
johnbillion opened this issue Mar 6, 2019 · 1 comment
Open

Iterator that supports multiple items within files #56

johnbillion opened this issue Mar 6, 2019 · 1 comment
Assignees

Comments

@johnbillion
Copy link
Member

johnbillion commented Mar 6, 2019

Using a separate file for each individual exported item is fine until you get to really large numbers and want to batch multiple items into a file in order to reduce the overall number of files (for I/O performance purposes, for example).

I'm working on writing an iterator that supports a JSON encoded array within each file that gets iterated over by the Files iterator, including support for updating the progress bar correctly, and only passing one item to process_item() as expected.

Combined with #55 this allows for much more efficient use of files on the filesystem.

@johnbillion johnbillion self-assigned this Mar 6, 2019
@johnbillion johnbillion changed the title Iterator that supports multiple items with files Iterator that supports multiple items within files Mar 11, 2019
@tcrsavage
Copy link
Contributor

@johnbillion I want to pull out the progress indicator moving forward. It's not particularly useful for a developer to see a progress bar when that information can be contained within a line by line debug e.g:

1/1000 Processing item source ID: 123
1/1000 Inserted item ID: 456
2/1000 Processing item source ID: 789
2/1000 Warning: Unable to insert item 789

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants