refactor: revisit transfer samples (#140)

* extracting negotiation tests * implementing Transfer02consumerPullTest * implementing Transfer03providerPushTest.java * implementing Transfer04eventConsumerTest.java * moving open telemetry to 'advanced' directory * reworking download of open telemetry jar inside gradle build file * checkStyle fixes * fix: clean up and merge * fix: added libs.edc.control.plane.api to build file * fix: fixed event consumer test * fix: resolving pr comments * refactor: move http-request-logger-wrapper.yaml to test directory * refactor: using docker image instead of docker compose for http request logger container * fix: fixing check style issues * fix: removing unnecessary stuff * refactor: removing HttpRequestLoggerUtil.java * fix: setting http-request-logger.jar archiveFilename explicitly
eclipse-edc · Nov 6, 2023 · e9e41d5 · e9e41d5
1 parent 3c7ff03
commit e9e41d5
Show file tree

Hide file tree

Showing 105 changed files with 1,909 additions and 3,097 deletions.
diff --git a/README.md b/README.md
@@ -42,6 +42,13 @@ EDC and run different transfer scenarios. Click the link above to learn about th
 
 All transfer samples are located in the `transfer` directory.
 
+### [Advanced](./advanced/README.md)
+
+Collection of advanced topics regarding the EDC framework.
+Click the link above to learn about the transfer samples in more detail.
+
+All transfer samples are located in the `advanced` directory.
+
 ## Contributing
 
 See [how to contribute](CONTRIBUTING.md).

diff --git a/advanced/README.md b/advanced/README.md
@@ -0,0 +1,12 @@
+# Advanced samples
+
+The samples in this scope teach advanced topics about the EDC framework. 
+
+> Before starting with these samples, be sure to check out the [basic](../basic/README.md) and [transfer](../transfer/README.md) samples!
+
+## Samples
+
+### [Open Telemetry 01](./advanced-01-open-telemetry/README.md) Telemetry with OpenTelemetry and Micrometer
+
+In this sample you will learn how to generate traces with [OpenTelemetry](https://opentelemetry.io) 
+and collect and visualize them with [Jaeger](https://www.jaegertracing.io/).
diff --git a/...sfer/transfer-04-open-telemetry/README.md → ...nced/advanced-01-open-telemetry/README.md b/...sfer/transfer-04-open-telemetry/README.md → ...nced/advanced-01-open-telemetry/README.md
@@ -1,11 +1,10 @@
 # Telemetry with OpenTelemetry and Micrometer
 
-This sample builds on top of [sample transfer-01-file-transfer](../transfer-01-file-transfer/README.md) to show how you
-can:
+This sample will show you how you can:
 
-- generate traces with [OpenTelemetry](https://opentelemetry.io) and collect and visualize them with [Jaeger](https://www.jaegertracing.io/).
+- generate traces with [OpenTelemetry](https://opentelemetry.io) and collect and visualize them with [Jaeger](https://www.jaegertracing.io/)
 - automatically collect metrics from infrastructure, server endpoints and client libraries with [Micrometer](https://micrometer.io)
-  and visualize them with [Prometheus](https://prometheus.io).
+  and visualize them with [Prometheus](https://prometheus.io)
 
 For this, this sample uses the Open Telemetry Java Agent, which dynamically injects bytecode to capture telemetry from
 several popular [libraries and frameworks](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation).
@@ -27,38 +26,91 @@ is configured to expose a Prometheus metrics endpoint.
 To run the consumer, the provider, and Jaeger execute the following commands in the project root folder:
 
 ```bash
-docker-compose -f transfer/transfer-04-open-telemetry/docker-compose.yaml up --abort-on-container-exit
+docker-compose -f advanced/advanced-01-open-telemetry/docker-compose.yaml up --abort-on-container-exit
 ```
 
-Once the consumer and provider are up, start a contract negotiation by executing:
+Open a new terminal.
+
+Register data planes for provider and consumer:
+
+```bash
+curl -H 'Content-Type: application/json' \
+  -H "X-Api-Key: password" \
+  -d @transfer/transfer-00-prerequisites/resources/dataplane/register-data-plane-provider.json \
+  -X POST "http://localhost:19193/management/v2/dataplanes" \
+  -s | jq
+```
 
 ```bash
-curl -X POST -H "Content-Type: application/json" -H "X-Api-Key: password" -d @transfer/transfer-04-open-telemetry/contractoffer.json "http://localhost:9192/management/v2/contractnegotiations"
+curl -H 'Content-Type: application/json' \
+  -H "X-Api-Key: password" \
+  -d @transfer/transfer-00-prerequisites/resources/dataplane/register-data-plane-consumer.json \
+  -X POST "http://localhost:29193/management/v2/dataplanes" \
+  -s | jq
 ```
 
-The contract negotiation causes an HTTP request sent from the consumer to the provider connector, followed by another
-message from the provider to the consumer connector. Query the status of the contract negotiation by executing the
-following command. Wait until the negotiation is in CONFIRMED state and note down the contract agreement id.
+Create an asset:
 
 ```bash
-curl -X GET -H 'X-Api-Key: password' "http://localhost:9192/management/v2/contractnegotiations/{UUID}"
+curl -H "X-Api-Key: password" \
+  -d @transfer/transfer-01-negotiation/resources/create-asset.json \
+  -H 'content-type: application/json' http://localhost:19193/management/v2/assets \
+  -s | jq
 ```
 
-Finally, update the contract agreement id in the `filetransfer.json` file and execute a file transfer with the following command:
+Create a Policy on the provider connector:
 
 ```bash
-curl -X POST -H "Content-Type: application/json" -H "X-Api-Key: password" -d @transfer/transfer-04-open-telemetry/filetransfer.json "http://localhost:9192/management/v2/transferprocesses"
+curl -H "X-Api-Key: password" \
+  -d @transfer/transfer-01-negotiation/resources/create-policy.json \
+  -H 'content-type: application/json' http://localhost:19193/management/v2/policydefinitions \
+  -s | jq
+```
+
+Follow up with the creation of a contract definition:
+
+```bash
+curl -H "X-Api-Key: password" \
+  -d @transfer/transfer-01-negotiation/resources/create-contract-definition.json \
+  -H 'content-type: application/json' http://localhost:19193/management/v2/contractdefinitions \
+  -s | jq
+```
+
+Start a contract negotiation:
+
+```bash
+curl -H "X-Api-Key: password" \
+  -H "Content-Type: application/json" \
+  -d @advanced/advanced-01-open-telemetry/resources/negotiate-contract.json \
+  -X POST "http://localhost:29193/management/v2/contractnegotiations" \
+  -s | jq
+```
+
+Wait until the negotiation is in `FINALIZED` state and call
+
+```bash
+curl -X GET -H 'X-Api-Key: password' "http://localhost:29193/management/v2/contractnegotiations/{UUID}"
+```
+to get the contract agreement id.
+
+Finally, update the contract agreement id in the [request body](resources/start-transfer.json) and execute a file transfer with the following command:
+
+```bash
+curl -H "X-Api-Key: password" \
+  -H "Content-Type: application/json" \
+  -d @advanced/advanced-01-open-telemetry/resources/start-transfer.json \
+  -X POST "http://localhost:29193/management/v2/transferprocesses"
 ```
 
 You can access the Jaeger UI on your browser at `http://localhost:16686`. In the search tool, we can select the service
 `consumer` and click on `Find traces`. A trace represents an event and is composed of several spans. You can inspect
 details on the spans contained in a trace by clicking on it in the Jaeger UI.
 
 Example contract negotiation trace:
-![Contract negotiation](./attachments/contract-negotiation-trace.png)
+![Contract negotiation](attachments/contract-negotiation-trace.png)
 
 Example file transfer trace:
-![File transfer](./attachments/file-transfer-trace.png)
+![File transfer](attachments/file-transfer-trace.png)
 
 OkHttp and Jetty are part of the [libraries and frameworks](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation)
 that OpenTelemetry can capture telemetry from. We can observe spans related to OkHttp and Jetty as EDC uses both
@@ -80,7 +132,7 @@ which has to be stored in the root folder of this sample as well. The only addit
   consumer:
     build:
       context: ../..
-      dockerfile: transfer/transfer-04-open-telemetry/open-telemetry-consumer/Dockerfile
+      dockerfile: advanced/advanced-01-open-telemetry/open-telemetry-consumer/Dockerfile
     volumes:
       - ./:/resources
     ports:
@@ -128,6 +180,4 @@ it, otherwise it will use the registered global OpenTelemetry. You can look at t
 [ServiceLoader documentation](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/ServiceLoader.html)
 to have more information about service providers.
 
----
-
-[Previous Chapter](../transfer-03-modify-transferprocess/README.md) | [Next Chapter](../transfer-05-file-transfer-cloud/README.md)
+---
diff --git a/...ttachments/contract-negotiation-trace.png → ...ttachments/contract-negotiation-trace.png b/...ttachments/contract-negotiation-trace.png → ...ttachments/contract-negotiation-trace.png
diff --git a/...metry/attachments/file-transfer-trace.png → ...metry/attachments/file-transfer-trace.png b/...metry/attachments/file-transfer-trace.png → ...metry/attachments/file-transfer-trace.png
diff --git a/advanced/advanced-01-open-telemetry/docker-compose.yaml b/advanced/advanced-01-open-telemetry/docker-compose.yaml
@@ -0,0 +1,93 @@
+version: "3.8"
+
+services:
+
+  consumer:
+    build:
+      context: ../..
+      dockerfile: advanced/advanced-01-open-telemetry/open-telemetry-consumer/Dockerfile
+    volumes:
+      - ./:/open-telemetry
+      - ../../transfer/transfer-00-prerequisites/:/prerequisites
+    ports:
+      - "29193:29193"
+      - "29194:29194"
+    environment:
+      EDC_HOSTNAME: consumer
+      OTEL_SERVICE_NAME: consumer
+      OTEL_TRACES_EXPORTER: jaeger
+      OTEL_EXPORTER_JAEGER_ENDPOINT: http://jaeger:14250
+      OTEL_METRICS_EXPORTER: prometheus
+      WEB_HTTP_PORT: 29191
+      WEB_HTTP_PATH: /api
+      WEB_HTTP_PUBLIC_PORT: 29291
+      WEB_HTTP_PUBLIC_PATH: /public
+      WEB_HTTP_CONTROL_PORT: 29192
+      WEB_HTTP_CONTROL_PATH: /control
+      WEB_HTTP_MANAGEMENT_PORT: 29193
+      WEB_HTTP_MANAGEMENT_PATH: /management
+      WEB_HTTP_PROTOCOL_PORT: 29194
+      WEB_HTTP_PROTOCOL_PATH: /protocol
+      EDC_CONTROL_ENDPOINT: http://consumer:29192/control
+      EDC_DSP_CALLBACK_ADDRESS: http://consumer:29194/protocol
+      EDC_PARTICIPANT_ID: consumer
+      EDC_API_AUTH_KEY: password
+      EDC_KEYSTORE: /prerequisites/resources/certs/cert.pfx
+      EDC_KEYSTORE_PASSWORD: 123456
+      EDC_VAULT: /prerequisites/resources/configuration/provider-vault.properties
+      EDC_FS_CONFIG: /prerequisites/resources/configuration/provider-configuration.properties
+    entrypoint: java
+      -javaagent:/app/opentelemetry-javaagent.jar
+      -Djava.util.logging.config.file=/open-telemetry/resources/logging.properties
+      -jar /app/connector.jar
+
+  provider:
+    build:
+      context: ../..
+      dockerfile: advanced/advanced-01-open-telemetry/open-telemetry-provider/Dockerfile
+    volumes:
+      - ./:/open-telemetry
+      - ../../transfer/transfer-00-prerequisites/:/prerequisites
+    ports:
+      - "19193:19193"
+      - "19192:19192"
+    environment:
+      EDC_HOSTNAME: provider
+      OTEL_SERVICE_NAME: provider
+      OTEL_TRACES_EXPORTER: jaeger
+      OTEL_EXPORTER_JAEGER_ENDPOINT: http://jaeger:14250
+      WEB_HTTP_PORT: 19191
+      WEB_HTTP_PATH: /api
+      WEB_HTTP_PUBLIC_PORT: 19291
+      WEB_HTTP_PUBLIC_PATH: /public
+      WEB_HTTP_CONTROL_PORT: 19192
+      WEB_HTTP_CONTROL_PATH: /control
+      WEB_HTTP_MANAGEMENT_PORT: 19193
+      WEB_HTTP_MANAGEMENT_PATH: /management
+      WEB_HTTP_PROTOCOL_PORT: 19194
+      WEB_HTTP_PROTOCOL_PATH: /protocol
+      EDC_CONTROL_ENDPOINT: http://provider:19192/control
+      EDC_DSP_CALLBACK_ADDRESS: http://provider:19194/protocol
+      EDC_PARTICIPANT_ID: provider
+      EDC_API_AUTH_KEY: password
+      EDC_KEYSTORE: /prerequisites/resources/certs/cert.pfx
+      EDC_KEYSTORE_PASSWORD: 123456
+      EDC_VAULT: /prerequisites/resources/configuration/consumer-vault.properties
+      EDC_FS_CONFIG: /prerequisites/resources/configuration/consumer-configuration.properties
+      EDC_SAMPLES_TRANSFER_01_ASSET_PATH: /open-telemetry/README.md
+    entrypoint: java
+      -javaagent:/app/opentelemetry-javaagent.jar
+      -Djava.util.logging.config.file=/open-telemetry/resources/logging.properties
+      -jar /app/connector.jar
+
+  jaeger:
+    image: jaegertracing/all-in-one
+    ports:
+      - "16686:16686"
+
+  prometheus:
+    image: prom/prometheus:v2.30.3
+    volumes:
+      - ./prometheus/:/etc/prometheus/
+    ports:
+      - "9090:9090"
diff --git a/advanced/advanced-01-open-telemetry/open-telemetry-consumer/Dockerfile b/advanced/advanced-01-open-telemetry/open-telemetry-consumer/Dockerfile
@@ -0,0 +1,11 @@
+FROM gradle:jdk17 AS build
+
+WORKDIR /home/gradle/project/
+COPY --chown=gradle:gradle . /home/gradle/project/
+RUN gradle advanced:advanced-01-open-telemetry:open-telemetry-consumer:build
+
+FROM openjdk:17-slim
+
+WORKDIR /app
+COPY --from=build /home/gradle/project/advanced/advanced-01-open-telemetry/open-telemetry-consumer/build/libs/opentelemetry-javaagent-*.jar /app/opentelemetry-javaagent.jar
+COPY --from=build /home/gradle/project/advanced/advanced-01-open-telemetry/open-telemetry-consumer/build/libs/consumer.jar /app/connector.jar
diff --git a/.../open-telemetry-consumer/build.gradle.kts → .../open-telemetry-consumer/build.gradle.kts b/.../open-telemetry-consumer/build.gradle.kts → .../open-telemetry-consumer/build.gradle.kts
@@ -26,19 +26,29 @@ plugins {
 dependencies {
 
     implementation(libs.edc.control.plane.api.client)
+    implementation(libs.edc.control.plane.api)
     implementation(libs.edc.control.plane.core)
-    implementation(libs.edc.data.plane.selector.core)
-    implementation(libs.edc.micrometer.core)
-    implementation(libs.edc.api.observability)
+
+    implementation(libs.edc.dsp)
     implementation(libs.edc.configuration.filesystem)
+    implementation(libs.edc.vault.filesystem)
+
     implementation(libs.edc.iam.mock)
-    implementation(libs.edc.auth.tokenbased)
     implementation(libs.edc.management.api)
-    implementation(libs.edc.dsp)
+    implementation(libs.edc.transfer.data.plane)
+    implementation(libs.edc.transfer.pull.http.receiver)
+
+    implementation(libs.edc.data.plane.selector.api)
+    implementation(libs.edc.data.plane.selector.core)
+    implementation(libs.edc.data.plane.selector.client)
+
+    implementation(libs.edc.data.plane.api)
+    implementation(libs.edc.data.plane.core)
+    implementation(libs.edc.data.plane.http)
+
+    implementation(libs.edc.api.observability)
+    implementation(libs.edc.auth.tokenbased)
 
-    runtimeOnly(libs.opentelemetry)
-    runtimeOnly(libs.edc.jersey.micrometer)
-    runtimeOnly(libs.edc.jetty.micrometer)
     runtimeOnly(libs.edc.monitor.jdk.logger)
 }
 
@@ -51,25 +61,17 @@ tasks.withType<com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar> {
     archiveFileName.set("consumer.jar")
 }
 
-tasks.register("copyOpenTelemetryJar") {
-    doLast {
-        val file = file("../opentelemetry-javaagent.jar")
-
-        if (!file.exists()) {
-            sourceSets["main"]
-                    .runtimeClasspath
-                    .files
-                    .find { it.name.contains("opentelemetry-javaagent") }
-                    ?.path
-                    ?.let {
-                        val sourcePath = Paths.get(it)
-                        val targetPath = Paths.get(file.path)
-                        Files.copy(sourcePath, targetPath, StandardCopyOption.REPLACE_EXISTING)
-                    }
-        }
+tasks.register("copyOpenTelemetryJar", Copy::class) {
+    val openTelemetry = configurations.create("open-telemetry")
+
+    dependencies {
+        openTelemetry(libs.opentelemetry)
     }
+
+    from(openTelemetry)
+    into("build/libs")
 }
 
 tasks.build {
     finalizedBy("copyOpenTelemetryJar")
-}
+}
diff --git a/advanced/advanced-01-open-telemetry/open-telemetry-provider/Dockerfile b/advanced/advanced-01-open-telemetry/open-telemetry-provider/Dockerfile
@@ -0,0 +1,11 @@
+FROM gradle:jdk17 AS build
+
+WORKDIR /home/gradle/project/
+COPY --chown=gradle:gradle . /home/gradle/project/
+RUN gradle advanced:advanced-01-open-telemetry:open-telemetry-provider:build
+
+FROM openjdk:17-slim
+
+WORKDIR /app
+COPY --from=build /home/gradle/project/advanced/advanced-01-open-telemetry/open-telemetry-provider/build/libs/opentelemetry-javaagent-*.jar /app/opentelemetry-javaagent.jar
+COPY --from=build /home/gradle/project/advanced/advanced-01-open-telemetry/open-telemetry-provider/build/libs/provider.jar /app/connector.jar