You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using a separate file for each individual exported item is fine until you get to really large numbers and want to batch multiple items into a file in order to reduce the overall number of files (for I/O performance purposes, for example).
I'm working on writing an iterator that supports a JSON encoded array within each file that gets iterated over by the Files iterator, including support for updating the progress bar correctly, and only passing one item to process_item() as expected.
Combined with #55 this allows for much more efficient use of files on the filesystem.
The text was updated successfully, but these errors were encountered:
@johnbillion I want to pull out the progress indicator moving forward. It's not particularly useful for a developer to see a progress bar when that information can be contained within a line by line debug e.g:
Using a separate file for each individual exported item is fine until you get to really large numbers and want to batch multiple items into a file in order to reduce the overall number of files (for I/O performance purposes, for example).
I'm working on writing an iterator that supports a JSON encoded array within each file that gets iterated over by the Files iterator, including support for updating the progress bar correctly, and only passing one item to
process_item()
as expected.Combined with #55 this allows for much more efficient use of files on the filesystem.
The text was updated successfully, but these errors were encountered: