PluckInBatches

ActiveRecord comes with find_each / find_in_batches / in_batches methods to batch process records from a database. ActiveRecord also has the pluck method which allows the selection of a set of fields without pulling the entire record into memory.

This gem combines these ideas and provides pluck_each and pluck_in_batches methods to allow batch processing of plucked fields from the database.

It performs half of the number of SQL queries, allocates up to half of the memory and is up to 2x faster (or more, depending on how far is your database from the application) than the available alternative:

# Before
User.in_batches do |batch| # or .find_in_batches, or .select(:email).find_each etc
  emails = batch.pluck(:emails)
  # do something with emails
end

# Now, using this gem (up to 2x faster)
User.pluck_in_batches(:email) do |emails|
  # do something with emails
end

Note: You may also find sidekiq-iteration useful when iterating over large collections in Sidekiq jobs.

Requirements

• Ruby 2.7+
• ActiveRecord 6+

If you need support for older versions, open an issue.

Installation

Add this line to your application's Gemfile:

gem 'pluck_in_batches'

And then execute:

$ bundle

Or install it yourself as:

$ gem install pluck_in_batches

Usage

pluck_each

Behaves similarly to find_each ActiveRecord's method, but yields each set of values corresponding to the specified columns.

# Single column
User.where(active: true).pluck_each(:email) do |email|
  # do something with email
end

# Multiple columns
User.where(active: true).pluck_each(:id, :email) do |id, email|
  # do something with id and email
end

pluck_in_batches

Behaves similarly to in_batches ActiveRecord's method, but yields each batch of values corresponding to the specified columns.

# Single column
User.where("age > 21").pluck_in_batches(:email) do |emails|
  jobs = emails.map { |email| PartyReminderJob.new(email) }
  ActiveJob.perform_all_later(jobs)
end

# Multiple columns
User.pluck_in_batches(:name, :email).with_index do |group, index|
  puts "Processing group ##{index}"
  jobs = group.map { |name, email| PartyReminderJob.new(name, email) }
  ActiveJob.perform_all_later(jobs)
end

# Custom arel column
User.pluck_in_batches(:id, Arel.sql("json_extract(users.metadata, '$.rank')")).with_index do |group, index|
  # ...
end

Both methods support the following configuration options:

• :batch_size - Specifies the size of the batch. Defaults to 1000. Also aliased as :of.
• :start - Specifies the primary key value to start from, inclusive of the value.
• :finish - Specifies the primary key value to end at, inclusive of the value.
• :error_on_ignore - Overrides the application config to specify if an error should be raised when an order is present in the relation.
• :cursor_column - Specifies the column(s) on which the iteration should be done. This column(s) should be orderable (e.g. an integer or string). Defaults to primary key.
• :order - Specifies the primary key order (can be :asc or :desc or an array consisting of :asc or :desc). Defaults to :asc.

Development

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and the created tag, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/fatkodima/pluck_in_batches.

License

The gem is available as open source under the terms of the MIT License.