Skip to content

Commit

Permalink
Updating slides.
Browse files Browse the repository at this point in the history
  • Loading branch information
philipnorton42 committed Oct 17, 2024
1 parent 1ca3880 commit 1405c53
Show file tree
Hide file tree
Showing 3 changed files with 161 additions and 39 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/assets/images/lily58.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
200 changes: 161 additions & 39 deletions src/slides.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ style: |
p.centre {
text-align: center;
}
footer: "Philip Norton [hashbangcode.com](https://www.hashbangcode.com) [@hashbangcode](https://twitter.com/hashbangcode) [@philipnorton42](https://twitter.com/philipnorton42)"
footer: "Philip Norton [hashbangcode.com](https://www.hashbangcode.com) [@hashbangcode](https://fosstodon.org/@hashbangcode) [@philipnorton42](https://fosstodon.org/@philipnorton42)"
marp: true
---

Expand All @@ -28,8 +28,9 @@ marp: true

# Philip Norton
- Developer at Code Enigma
- Write articles at `#! code` www.hashbangcode.com
- Organiser of NWDUG
- Writer at `#! code` (www.hashbangcode.com)
- NWDUG host
![bg h:50% right:40%](../src/assets/images/lily58.png)

---

Expand All @@ -42,6 +43,16 @@ marp: true

---

# The Drupal Batch API

<!--
The Drupal Batch API is a system that allows us to split up large tasks into smaller chunks so that we don't overload the web server. This is a really important concept to understand if you are developing your own Drupal modules, especially if you want to allow users to perform actions on lots of items at once.
In this session we will look at why we need the Drupal Batch API and how it prevents issues. We'll then do a dive into the code surrounding the Batch API and how to run your own batch operations.
There will be a few demos on running the Batch API to perform some simple tasks.
Finally, we'll look at how Drupal uses the Batch API internally and some use cases of the Batch API in contributed modules.
-->
---

## The Batch API
Allows data to be processed in small chunks in order to prevent timeout errors or memory problems.

Expand All @@ -59,36 +70,38 @@ Allows data to be processed in small chunks in order to prevent timeout errors o


<!--
https://www.cloudflare.com/learning/performance/more/website-performance-conversion-rates/
Source: https://www.cloudflare.com/learning/performance/more/website-performance-conversion-rates/
-->

---

## Server Timeouts

- Servers are designed to throw errors is something takes too long.
- Servers are designed to throw errors is something takes too long. Some defaults:

- PHP (`max_execution_time`) - 30 seconds
- PHP (`memory_limit`) - 256MB (recommended for Drupal)
- Apache (`TimeOut`) - 60 seconds
- Nginx (`send_timeout`/`proxy_send_timeout`) - 60 seconds

<!--
You can override all these options.
But, setting them high would cause problems on your web server.
Think about how long a page request should take in you web server.
- You can override all these options.
- But, setting them high would cause problems on your web server.
- Think about how long a page request should take in you web server.
2-5 seconds MAX!
- The more resources you give to your web server, the users you can accommodate at once.
-->

---

## What Problem Are We Solving?
## The Problem

- Trying to do too much in one page request.
- Downloading lots of data from an api.
- Create/update/delete lots of entities.
<br>
- The page times out or runs out of memory
- Users assume page is broken and click away.
- The page times out or runs out of memory.

---
<!-- _footer: "" -->
Expand All @@ -97,7 +110,7 @@ Think about how long a page request should take in you web server.
- Solves these problems by splitting long tasks into smaller chunks.
- Drupal then runs them through a special interface.

![h:10%](../src/assets/images/batch_process_running.png)
![left:center](../src/assets/images/batch_process_running.png)

---

Expand All @@ -107,7 +120,7 @@ Think about how long a page request should take in you web server.

## The Batch Process

The batch process has the following tasks:
The Batch API can be thought of as the following:

- <strong>Initialise</strong> - Set up the batch run, define callbacks.
- <strong>Process</strong> - The batch process operations.
Expand Down Expand Up @@ -159,37 +172,43 @@ foreach ($chunks as $id => $chunk) {

---

## Process - batch_set()
## Process

Set the batch running by calling `toArray()` and passing the array to `batch_set()`.

```php
batch_set($batch->toArray());
```

The whole purpose of BatchBuilder is to generate that array.
The whole purpose of `BatchBuilder` is to generate that array.
This will trigger and start up the batch process.
<!--
- This will redirect to the batch processor interface and call the process operations.
- Yes you can also just define the array and send it to batch_set(), but I find BatchBuilder a nicer interface.
-->

---
<!-- _footer: "" -->
## Process

- Callback defined in the `addOperation()` method.
- The callbacks defined in the `addOperation()` method.
- Parameters are the array of arguments you set.
- `$context` is passed by the Batch processor and is used to track progress.
- `$context` is passed as the last parameter is used to track progress.

```php
public static function batchProcess(int $batchId, array $chunk, array &$context): void {
}
```

- This method is called multiple times (depending on the batch run).

---

<!-- _footer: "" -->
## Process - Tracking Progress

- The `$context` parameter is an array that is maintained between different batch calls.
- The `"sandbox"` element is used inside the batch process and is deleted at the end of the batch run.
- The `"results"` element is will be passed to the finished callback and is often used to track progres.
- The `"results"` element is will be passed to the finished callback and is often used to track progres for reporting.

```php
public static function batchProcess(int $batchId, array $chunk, array &$context): void {
Expand All @@ -203,7 +222,7 @@ public static function batchProcess(int $batchId, array $chunk, array &$context)
## Process - Messages

- As the batch runs you can set a `"message"` element to print messages to the user.
- This will appaer above the batch progress bar.
- This will appaer above the batch progress bar as the batch progresses.

```php
// Message above progress bar.
Expand All @@ -230,18 +249,44 @@ public static function batchFinished(
}
```
<!--
$success - TRUE if all batch API tasks were completed successfully.
$results - An results array from the batch processing operations.
$operations - A list of the operations that had not been completed.
$elapsed - Batch.inc kindly provides the elapsed processing time in seconds.
- $success - TRUE if all batch API tasks were completed successfully.
- $results - An results array from the batch processing operations.
- $operations - A list of the operations that had not been completed.
- $elapsed - Batch.inc kindly provides the elapsed processing time in seconds.
-->
---

## Batch Internal Workings
## Finished - The Finished Callback

- The Batch API is really an extension of the Queue system.
- When you add operations to the batch you are adding items to the queue.
- The Drupal batch runner then pulls items out of the queue and feeds them to the process method.
For example, you might want to report the results of the batch run to your user.

```php
public static function batchFinished(bool $success, array $results, array $operations, string $elapsed): void {
$messenger = \Drupal::messenger();
if ($success) {
$messenger->addMessage(t('@process processed @count, skipped @skipped, updated @updated, failed @failed in @elapsed.', [
'@process' => $results['process'],
'@count' => $results['progress'],
'@skipped' => $results['skipped'],
'@updated' => $results['updated'],
'@failed' => $results['failed'],
'@elapsed' => $elapsed,
]));
}
}
```

---

<!-- _footer: "" -->
## The Running Batch

![width:30cm center](../src/assets/images/batch_process_running_requests.png)

<!--
- The batch JavaScript called the /batch endpoint.
- This handles the processing of the batch operations.
-->

---

Expand Down Expand Up @@ -276,7 +321,6 @@ This is run over and over until we issue the finished state.

---


## The Batch "finished" State

It is common to divide the progress by the maximum number of items.
Expand All @@ -289,13 +333,13 @@ $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['ma
<!-- _footer: "" -->
## The Batch "finished" State

This means we can just launch the batch with no arguments.
This also means that we can just launch the batch with no arguments.

```php
$batch->addOperation([BatchProcessNodes::class, 'batchProcess']);
```

The "max" is found in the batchProcess() method.
The `max` property is found in the batchProcess() method the first time it is run.

```php
public static function batchProcess(array &$context): void {
Expand All @@ -307,6 +351,20 @@ public static function batchProcess(array &$context): void {
}
```

---

## Batch Internal Workings

- The Batch API is really an extension of the Queue system.
- When you add operations to the batch you are adding items to the queue.
- The Drupal batch runner then pulls items out of the queue and feeds them to the process method.

---

## When To Use The Batch API

- If the request processes lots of items them move it into a batch.
- Use the batch system early to save having to rework things later.

---

Expand All @@ -328,16 +386,17 @@ This will run the batch on the command line.

---

## When To Use The Batch API
## Drush

- If the request processes lots of items them move it into a batch.
- Use the batch system early to save having to rework things later.
- Be careful! Drush will process the batch operations in the same memory space.
- As you are on the command line you won't time out, but you can run out of memory.

---

# Examples Of Batch API In Action

- Some live demos!
Some live demos!

---

## Batch Using A Form
Expand All @@ -355,14 +414,64 @@ This will run the batch on the command line.

---

# Batch Inside Drupal
# The Batch API Inside Drupal

---

## The Update Hook

- Update hooks get a $sandbox variable. This is actually a batch $context array.
- Update hooks get a `$sandbox` variable. This is actually a batch `$context` array.
- You can set the `#finished` property in the `$sandbox` array to stop the batch.

---
<!-- _footer: "" -->
## The Update Hook
An example of a batched update hook.
```php
function batch_update_example_update_10001(&$sandbox) {
if (!isset($sandbox['progress'])) {
$sandbox['progress'] = 0;
$sandbox['max'] = 1000;
}

for ($i = 0; $i < 100; $i++) {
// Keep track of progress.
$sandbox['progress']++;
// Do some actions...
}
\Drupal::messenger()->addMessage($sandbox['progress'] . ' items processed.');

$sandbox['#finished'] = $sandbox['progress'] / $sandbox['max'];
}
```

---

## General Batch Uses

- Drupal also makes use of the Batch API in lots of different situations. For example:
- Installing Drupal.
- Deleting users.
- Bulk content updates.
- Installing modules.
- Importing translations.
- Importing configuration.
- And much more!

---
<!-- _footer: "" -->

## Top Tips

- If the data needs to be processed in real time then use a batch; otherwise use a standard queue.
- Kick off your batches in a form or controller, but process the batch in a separate class. This allows easy Drush integration.
- Use the `finished` property to make dynamic batches; rather than preloaded.
- Keep your batch operations simple. Breat them apart into separate operations if needed.
- Allow batch operations to pick up where they left off.

<!--
- Think about the footprint of your batch operations. Keep them small.
-->
---

# Modules That Use Batch
Expand All @@ -386,6 +495,22 @@ This will run the batch on the command line.

---

## Views Data Export

- A Views plugin that exports data in a number of different formats.

<small>https://www.drupal.org/project/views_data_export</small>

---

## Batch Plugin

- Wraps the Batch API in a plugin to make your batch operations pluggable.

<small>https://www.drupal.org/project/batch_plugin</small>

---

## Resources

- [Drupal 11: An Introduction To Batch Processing With The Batch API](https://www.hashbangcode.com/article/drupal-11-introduction-batch-processing-batch-api)
Expand All @@ -409,6 +534,3 @@ This will run the batch on the command line.
- Slides: https://github.com/hashbangcode/drupal-batch-api-talk

![bg h:50% right:40%](../src/assets/images/qr_slides.png)

---

0 comments on commit 1405c53

Please sign in to comment.