You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
subTitle: Execution Context - Why to have an execution context & what it does
5
-
---
5
+
width: large
6
+
---
7
+
8
+
## Execution Context - Why to have an execution context & what it does
9
+
10
+
In most of our examples our chain had access to the whole file system.
11
+
This means having multiple chains running together, or having a list of files each execution has generated is impossible.
12
+
13
+
Both the 🎵 Symfony Bundle(and therefore the 🦢 Sylius integration) and the Magento2 Module will use contextual chains.
14
+
This means the "main" operations have only access to a particular directory created for the execution of the chain.
15
+
16
+
Additional operations such as the ExternalFileFinderOperation and ExternalFileProcessor will be use to
17
+
process files that are either on a remote directory (sftp, bucket s3...) or files that are on the local file system.
18
+
Because operations such as the CsvLoader will not have access to those files unless they are copied into the contextual directory of the current execution.
19
+
20
+
Let start by a simple example.
21
+
22
+
### Write the result of an API to a CSV File.
23
+
24
+
{% capture description %}
25
+
For this we will first create a new ContextFactory using PerExecutionContextFactory.
26
+
This context factory will create unique contexts for each execution. This means a unique directory to run the etl
27
+
in; and a unique logger.
28
+
29
+
This is only needed if you are running the etl in **🐘 standalone**. With any integration this should be automatically $
30
+
handled for you. This chapter will be the last one where we do mention standalone integrations.
31
+
32
+
33
+
{% endcapture %}
34
+
{% capture code %}
35
+
```php
36
+
<?php
37
+
$workdir = __DIR__ . "/var/";
38
+
$dirManager = new ChainWorkDirManager($workdir);
39
+
$loggerFactory = new NullLoggerFactory();
40
+
$fileFactory = new LocalFileSystemFactory($dirManager);
41
+
42
+
return new PerExecutionContextFactory(
43
+
$dirManager,
44
+
$fileFactory,
45
+
$loggerFactory
46
+
);
47
+
```
48
+
{% endcapture %}
49
+
{% include block/etl-step.html code=code description=description %}
50
+
51
+
{% capture description %}
52
+
The execution is identified with objects of type ExecutionInterface set on the processor:
53
+
{% endcapture %}
54
+
{% capture code %}
55
+
```php
56
+
$options = [
57
+
'etl' => [
58
+
'execution' => new PockExecution(new DateTime())
59
+
]
60
+
];
61
+
62
+
$chainProcessor->process(
63
+
new ArrayIterator([[]]),
64
+
$options
65
+
);
66
+
```
67
+
{% endcapture %}
68
+
{% include block/etl-step.html code=code description=description %}
69
+
70
+
Executing this will create a directory in `var/` with the output result. Everytime you execute the chain a new
Copy file name to clipboardExpand all lines: docs/index.md
+80-2Lines changed: 80 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ subTitle:
7
7
## What is PHP-ETL
8
8
9
9
PHP-ETL is the go-to library for executing complex data import, export, and transformation tasks within PHP applications.
10
-
It offers seamless integrations with the [Symfony Framework](https://symfony.com/), [Sylius](https://sylius.com/fr/) , and can easily be extended to
10
+
It offers seamless integrations with the [🎵 Symfony Framework](https://symfony.com/), [🦢 Sylius](https://sylius.com/fr/) , and can easily be integrated to
11
11
other CMS and &frameworks, making it ideal for handling intricate data workflows with ease.
12
12
13
13
## Why PHP-ETL
@@ -29,6 +29,84 @@ PHP-ETL handles asynchronous operations—such as API calls—natively, allowing
29
29
like loading data into the database while making API calls. The library also supports visualizing data flows
30
30
through auto-generated diagrams, making complex workflows easier to understand and manage.
31
31
32
-
## A screenshot
32
+
## A execution tree
33
+
34
+
{% capture mermaid %}
35
+
flowchart TD
36
+
37
+
subgraph Execution
38
+
%% Nodes
39
+
0B(Extract Get Article API Params Data<br/><br/>2<iclass="sign in alternate icon"></i> / 2<iclass="sign out alternate icon"></i><br/>00:00.064<iclass="hourglass half icon"></i>)
40
+
style 0B fill:#EEE;
41
+
1B(Get products/articles until api stop's<br/><br/>2<iclass="sign in alternate icon"></i> / 2<iclass="sign out alternate icon"></i><br/>00:00.000<iclass="hourglass half icon"></i>)@{ shape: hex}
42
+
subgraph 1S[Get articles until api stop's]
43
+
100B(Make get Article API call<br/><br/>4<iclass="sign in alternate icon"></i> / 1<iclass="clock icon"></i> / 0<iclass="sign out alternate icon"></i><br/>00:05.243<iclass="hourglass half icon"></i>)
44
+
style 100B fill:#ffe294;
45
+
end
46
+
style 1B fill:#EEE;
47
+
2B(Write api response to file to keep history<br/><br/>4<iclass="sign in alternate icon"></i> / 4<iclass="sign out alternate icon"></i><br/>00:00.057<iclass="hourglass half icon"></i>)
48
+
style 2B fill:#EEE;
49
+
3B(Split response<br/><br/>5<iclass="sign in alternate icon"></i> / 5<iclass="sign out alternate icon"></i><br/>00:00.008<iclass="hourglass half icon"></i>)
50
+
style 3B fill:#EEE;
51
+
4B(Map Api fields with Sylius attributes code<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:01.482<iclass="hourglass half icon"></i>)
52
+
style 4B fill:#EEE;
53
+
5B(Branch to handle attribute option values & product imports<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>04:28.817<iclass="hourglass half icon"></i>)@{ shape: hex}
54
+
subgraph 5S[Branch to handle attribute option values & product imports]
55
+
500B(Split each attribute items<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:00.248<iclass="hourglass half icon"></i>)
56
+
style 500B fill:#EEE;
57
+
501B(Load Attribute from database<br/><br/>89571<iclass="sign in alternate icon"></i> / 89571<iclass="sign out alternate icon"></i><br/>00:46.995<iclass="hourglass half icon"></i>)
58
+
style 501B fill:#EEE;
59
+
502B(Add new choices to select attributes<br/><br/>89571<iclass="sign in alternate icon"></i> / 2<iclass="sign out alternate icon"></i><br/>00:09.363<iclass="hourglass half icon"></i>)
60
+
style 502B fill:#EEE;
61
+
503B(Persist attribute<br/><br/>2<iclass="sign in alternate icon"></i> / 2<iclass="sign out alternate icon"></i><br/>00:00.001<iclass="hourglass half icon"></i>)
62
+
style 503B fill:#EEE;
63
+
510B(Flush Doctrine before importing products<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:00.961<iclass="hourglass half icon"></i>)
64
+
style 510B fill:#EEE;
65
+
511B(Load Product from database<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:00.904<iclass="hourglass half icon"></i>)
66
+
style 511B fill:#EEE;
67
+
512B(Create or Update product<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:27.247<iclass="hourglass half icon"></i>)
68
+
style 512B fill:#EEE;
69
+
513B(Add price to product<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:01.651<iclass="hourglass half icon"></i>)
70
+
style 513B fill:#EEE;
71
+
514B(Persist entities<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:00.338<iclass="hourglass half icon"></i>)
72
+
style 514B fill:#EEE;
73
+
515B(Flush entities<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:02.117<iclass="hourglass half icon"></i>)
74
+
style 515B fill:#EEE;
75
+
516B(Clear doctrine<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:00.213<iclass="hourglass half icon"></i>)
76
+
style 516B fill:#EEE;
77
+
517B(Prepare data for Set association product API<br/><br/>2085<iclass="sign in alternate icon"></i> / 2085<iclass="sign out alternate icon"></i><br/>00:00.201<iclass="hourglass half icon"></i>)
78
+
style 517B fill:#EEE;
79
+
518B(Set Sylius Product ID association - API call<br/><br/>2085<iclass="sign in alternate icon"></i> / 2<iclass="sign out alternate icon"></i><br/>00:00.687<iclass="hourglass half icon"></i>)
80
+
style 518B fill:#EEE;
81
+
519B(Log association response<br/><br/>2085<iclass="sign in alternate icon"></i> / 4168<iclass="sign out alternate icon"></i><br/>00:00.012<iclass="hourglass half icon"></i>)
0 commit comments