diff --git a/federated-catalog/README.md b/federated-catalog/README.md new file mode 100644 index 00000000..0e9ad868 --- /dev/null +++ b/federated-catalog/README.md @@ -0,0 +1,40 @@ +# Federated Catalog Samples + +The samples in this section focus on the topic of Federated Catalogs. + +The Federated Catalog (FC) functions as an aggregated repository of +catalogs obtained from multiple +participants in the dataspace. To accomplish this, the FC utilizes crawlers +that periodically crawl the catalogs from each participant and store this list +of catalogs in a local cache. +By maintaining this locally cached version of catalogs, it eliminates the need to query +each participant individually, resulting in faster and more reliable queries. + + +The following samples shows how to +* implement, build and run different versions of FC e.g. + * standalone, + * embedded. + + +## Samples + +### [FC sample 00](./fc-00-basic/README.md): Federated Catalog Prerequisites +The purpose of this example is to make preparations for implementing Federated Catalog (FC). +We'll set up a basic federated catalog that includes necessary dependencies for triggering the FC, +along with some other modules to demonstrate FC functionalities. + +--- + +### Implement Different Versions of FC +### [FC sample 01](./fc-01-embedded/README.md): Implement an embedded federated catalog +This sample demonstrates how we can implement a federated catalog which is embedded in a connector. +The connector exposes a catalog endpoint that serves the consolidated list of catalogs. +### [FC sample 02](./fc-02-standalone/README.md): Implement a standalone federated catalog + +In this sample we focus on the implementation of +a standalone federated catalog. Unlike the previous sample, +a standalone federated catalog will not have the added functionalities of a connector. However, it also +exposes a catalog API that serves the list of catalogs. + +--- diff --git a/federated-catalog/fc-00-basic/README.md b/federated-catalog/fc-00-basic/README.md new file mode 100644 index 00000000..13cd65f1 --- /dev/null +++ b/federated-catalog/fc-00-basic/README.md @@ -0,0 +1,97 @@ +# Federated Catalog Prerequisites + +The purpose of this example is to make preparations for implementing Federated Catalog (FC) +and set up additional requirements for testing the FC functionalities. +For this purpose, we will be covering the following in this scope. +* `federated-catalog-base`: A basic federated catalog that includes necessary dependencies for triggering the FC. +* `participant-connector`: A connector with a contract offer. We will require this for testing the functionalities of the FCs in our later samples. +* `fixed-node-resolver`: A mock node directory resolver. +It provides a fixed Target Catalog Node, which represents the `participant-connector`. + + + +### federated-catalog-base +The [federated-catalog-base](../fc-00-basic/federated-catalog-base) will be used as a foundational module in our upcoming samples to trigger the FC. +It provides a [build.gradle.kts](./federated-catalog-base/build.gradle.kts) file that includes only the dependencies +essential for FC, without any additional functionality. +```shell +... +dependencies { + implementation(libs.edc.fc.spi.crawler) + runtimeOnly(libs.fc.core) + runtimeOnly(libs.fc.ext.api) +} +... +``` +Any further dependencies will be added in the later samples based on their use cases. + + +### fixed-node-resolver +The Federated Catalog requires a list of Target Catalog Nodes, which are essentially the DSP endpoints of the dataspace participants. +The catalog crawler then crawls these listed endpoints to collect their offered catalogs. +This list of Target Nodes is resolved by a Catalog Node Resolver which implements the [TargetNodeDirectory](https://github.com/eclipse-edc/FederatedCatalog/blob/main/spi/crawler-spi/src/main/java/org/eclipse/edc/crawler/spi/TargetNodeDirectory.java). +Check out [eclipse-edc/FederatedCatalog](https://github.com/eclipse-edc/FederatedCatalog/tree/main) for further information on this topic. + + +In this module, we've included a fixed Node Resolver, [fixed-node-resolver](./fixed-node-resolver) +that simply returns a hard-coded Target Node of the `participant-connector`. +However, we will not cover the implementation of the resolver in this sample; that will be explained in detail in later samples. + + +The purpose of including this [`fixed-node-resolver`](./fixed-node-resolver) +as a prerequisite, is the fact that we need to have some form of Target Node Resolver to demonstrate the functionality +of the federated catalogs that we are going to build in sample +[fc-01-embedded](../fc-01-embedded) and [fc-02-standalone](../fc-02-standalone). + +### participant connector + +When the federated catalog boots up, the crawler begins periodically invoking the Target Nodes returned by the +Node Resolver and collecting the catalogs offered by these nodes. To test whether our federated catalogs +(which we will build in later samples: [fc-01-embedded](../fc-01-embedded) and [fc-02-standalone](../fc-02-standalone)) can successfully request and retrieve these catalogs, we need at least one connector with a contract offer. + +Therefore, in this section, we will start a connector and then create a contract +for this connector. In the future samples, we will refer to it as `participant-connector`. +This `participant-connector` will function as a provider. +We will use the resources from the [transfer](../../transfer) sample to set up this connector. In the rest of this section we will, +* run the `participant-connector` +* create an asset for this `participant-connector` +* create a policy +* create a contract offer + +Although these topics were covered in the [transfer](../../transfer) section, we’ll document all the necessary commands here for easier execution. + + +#### Build connector jar +Use the following command to build a connector jar. +```shell +./gradlew transfer:transfer-00-prerequisites:connector:build +``` +#### Run the connector +Execute the following to run the connector jar. +```shell +java -Dedc.keystore=transfer/transfer-00-prerequisites/resources/certs/cert.pfx -Dedc.keystore.password=123456 -Dedc.fs.config=transfer/transfer-00-prerequisites/resources/configuration/provider-configuration.properties -jar transfer/transfer-00-prerequisites/connector/build/libs/connector.jar +``` + +--- + +Once the connector is running, carry out the commands below in sequence. +#### Create an asset +```shell +curl -d @transfer/transfer-01-negotiation/resources/create-asset.json \ + -H 'content-type: application/json' http://localhost:19193/management/v3/assets \ + -s | jq +``` + +#### Create a policy +```bash +curl -d @transfer/transfer-01-negotiation/resources/create-policy.json \ + -H 'content-type: application/json' http://localhost:19193/management/v3/policydefinitions \ + -s | jq +``` + +#### Create a contract definition +```bash +curl -d @transfer/transfer-01-negotiation/resources/create-contract-definition.json \ + -H 'content-type: application/json' http://localhost:19193/management/v3/contractdefinitions \ + -s | jq +``` \ No newline at end of file diff --git a/federated-catalog/fc-00-basic/federated-catalog-base/build.gradle.kts b/federated-catalog/fc-00-basic/federated-catalog-base/build.gradle.kts new file mode 100644 index 00000000..bbbffbda --- /dev/null +++ b/federated-catalog/fc-00-basic/federated-catalog-base/build.gradle.kts @@ -0,0 +1,26 @@ +/* + * Copyright (c) 2024 Fraunhofer-Gesellschaft + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Fraunhofer-Gesellschaft - initial API and implementation + * + */ + +plugins { + `java-library` + id("application") +} + +dependencies { + implementation(libs.edc.fc.spi.crawler) + runtimeOnly(libs.edc.fc.core) + runtimeOnly(libs.edc.fc.ext.api) +} + + diff --git a/federated-catalog/fc-00-basic/fixed-node-resolver/build.gradle.kts b/federated-catalog/fc-00-basic/fixed-node-resolver/build.gradle.kts new file mode 100644 index 00000000..7985a3f1 --- /dev/null +++ b/federated-catalog/fc-00-basic/fixed-node-resolver/build.gradle.kts @@ -0,0 +1,23 @@ +/* + * Copyright (c) 2024 Fraunhofer-Gesellschaft + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Fraunhofer-Gesellschaft - initial API and implementation + * + */ + +plugins { + `java-library` + id("application") +} + +dependencies { + implementation(libs.edc.fc.spi.crawler) +} + diff --git a/federated-catalog/fc-00-basic/fixed-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectory.java b/federated-catalog/fc-00-basic/fixed-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectory.java new file mode 100644 index 00000000..0424b986 --- /dev/null +++ b/federated-catalog/fc-00-basic/fixed-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectory.java @@ -0,0 +1,41 @@ +/* + * Copyright (c) 2024 Fraunhofer-Gesellschaft + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Fraunhofer-Gesellschaft - initial API and implementation + * + */ + +package org.eclipse.edc.sample.extension.fc; + +import org.eclipse.edc.crawler.spi.TargetNode; +import org.eclipse.edc.crawler.spi.TargetNodeDirectory; + +import java.util.ArrayList; +import java.util.List; + +public class CatalogNodeDirectory implements TargetNodeDirectory { + + @Override + public List getAll() { + List protocolList = new ArrayList<>(); + protocolList.add("dataspace-protocol-http"); + + TargetNode participantNode = new TargetNode("https://w3id.org/edc/v0.0.1/ns/", + "provider", + "http://localhost:19194/protocol", protocolList); + + return List.of(participantNode); + } + + @Override + public void insert(TargetNode targetNode) { + + } +} diff --git a/federated-catalog/fc-00-basic/fixed-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectoryExtension.java b/federated-catalog/fc-00-basic/fixed-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectoryExtension.java new file mode 100644 index 00000000..38ea1708 --- /dev/null +++ b/federated-catalog/fc-00-basic/fixed-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectoryExtension.java @@ -0,0 +1,28 @@ +/* + * Copyright (c) 2024 Fraunhofer-Gesellschaft + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Fraunhofer-Gesellschaft - initial API and implementation + * + */ + +package org.eclipse.edc.sample.extension.fc; + +import org.eclipse.edc.crawler.spi.TargetNodeDirectory; +import org.eclipse.edc.runtime.metamodel.annotation.Provider; +import org.eclipse.edc.spi.system.ServiceExtension; + +public class CatalogNodeDirectoryExtension implements ServiceExtension { + + @Provider + public TargetNodeDirectory federatedCacheNodeDirectory() { + return new CatalogNodeDirectory(); + } + +} diff --git a/federated-catalog/fc-00-basic/fixed-node-resolver/src/main/resources/META-INF/services/org.eclipse.edc.spi.system.ServiceExtension b/federated-catalog/fc-00-basic/fixed-node-resolver/src/main/resources/META-INF/services/org.eclipse.edc.spi.system.ServiceExtension new file mode 100644 index 00000000..df3d2d3d --- /dev/null +++ b/federated-catalog/fc-00-basic/fixed-node-resolver/src/main/resources/META-INF/services/org.eclipse.edc.spi.system.ServiceExtension @@ -0,0 +1 @@ +org.eclipse.edc.sample.extension.fc.CatalogNodeDirectoryExtension \ No newline at end of file diff --git a/federated-catalog/fc-01-embedded/README.md b/federated-catalog/fc-01-embedded/README.md new file mode 100644 index 00000000..7e6d3e1e --- /dev/null +++ b/federated-catalog/fc-01-embedded/README.md @@ -0,0 +1,123 @@ +# Embedded Federated Catalog + + +This sample demonstrates how we can implement a federated catalog which is embedded in a connector. +We will build one connector of such type and will call it `fc-connector`. +As discussed in the prerequisite sample [fc-00-basic](../fc-00-basic/README.md), +we will be using the Node Resolver +[fixed-node-resolver](../fc-00-basic/fixed-node-resolver) as a dependency in our embedded federated catalog. +Also, the `participant-connector` we set up in the previous sample should still be running. +This `participant-connector` will act as a provider, while the new `fc-connector` will act as a consumer. + + + +This sample will go through: + +* Implementation of an embedded FC +* Set up of the embedded FC, `fc-connector` +* Test catalog API endpoint of the `fc-connector` + + +### 1. Implementation the fc-connector +The [build.gradle.kts](../fc-01-embedded/fc-connector/build.gradle.kts) +file located in the [fc-01-embedded/fc-connector](../fc-01-embedded/fc-connector) +directory includes all the necessary dependencies for creating a connector, along with the `fc-00-basic:federated-catalog-base` +needed to trigger the FC. Additionally, we need to add `fc-00-basic:federated-catalog-base` as a dependency to enable the Catalog Node Resolver. + +```kotlin +dependencies { + runtimeOnly(project(":federated-catalog:fc-00-basic:federated-catalog-base")) + runtimeOnly(project(":federated-catalog:fc-00-basic:static-node-resolver")) +} +``` + +The [config.properties](../fc-01-embedded/fc-connector/config.properties) +file contains the necessary configurations +for this `fc-connector`, including the standard settings for a regular connector, along with additional configurations for a +federated catalog, such as catalog api endpoint and crawler execution interval. + +```properties +web.http.catalog.path=/api/catalog +web.http.catalog.port=29195 + +edc.catalog.cache.execution.delay.seconds=5 +edc.catalog.cache.execution.period.seconds=5 +edc.catalog.cache.partition.num.crawlers=5 +``` + +### 2. Start the fc-connector +#### Build the fc-connector JAR +Execute this command in project root to build the `fc-connector` JAR file: + +```bash +./gradlew federated-catalog:fc-01-embedded:fc-connector:build +``` + + +#### Run the fc-connector + +To run the connector, execute the following command + +```shell +java -Dedc.fs.config=federated-catalog/fc-01-embedded/fc-connector/config.properties -jar federated-catalog/fc-01-embedded/fc-connector/build/libs/fc-connector.jar +``` + +If the execution is successful, then the Catalog API of our `fc-connector` will listen on port `29195`. + +If you observe the logs, you can see the following recurring lines, + +> DEBUG 2024-11-14T13:53:48.472700883 [ExecutionManager] Run pre-execution task +> +>DEBUG 2024-11-14T13:53:48.494149928 [ExecutionManager] Loaded 1 work items from storage +> +>DEBUG 2024-11-14T13:53:48.495574504 [ExecutionManager] Crawler parallelism is 1, based on config and number of work items +> +>DEBUG 2024-11-14T13:53:48.497891576 [ExecutionManager] Crawler-f81f5514-5c7f-44aa-94bb-16998861789b: WorkItem acquired +> +>DEBUG 2024-11-14T13:53:48.790873233 [ExecutionManager] Crawler [Crawler-f81f5514-5c7f-44aa-94bb-16998861789b] is done + + +This means our FC crawler is running, and the crawler found one node, which is the `participant-connector` we had set up before. + + + +### 3. Test catalog query API + +To query the catalogs from `fc-connector` side, we can now call the catalog API of our embedded federated catalog. +Use the following request to invoke the catalog API: + +```http request +curl -d @federated-catalog/fc-01-embedded/resources/empty-query.json \ + -H 'content-type: application/json' http://localhost:29195/api/catalog/v1alpha/catalog/query \ + -s | jq +``` + +Sample output: +```json +[ + { + "@id": "a8a8cd64-269d-485c-8857-74d08b13ae3c", + "@type": "dcat:Catalog", + "dcat:dataset": { + "@id": "assetId", + "@type": "dcat:Dataset", + "odrl:hasPolicy": { + "@id": "MQ==:YXNzZXRJZA==:MjJmNDlhYTAtM2I3YS00ODkzLTkwZDctNTU5MTZhNmViOWJk" + }, + "dcat:distribution": [ + ], + "name": "product description", + "id": "assetId", + "contenttype": "application/json" + }, + "dcat:distribution": [], + "dcat:service": { + + }, + "dspace:participantId": "provider", + "originator": "http://localhost:19194/protocol", + "@context": { + } + } +] +``` \ No newline at end of file diff --git a/federated-catalog/fc-01-embedded/fc-connector/build.gradle.kts b/federated-catalog/fc-01-embedded/fc-connector/build.gradle.kts new file mode 100644 index 00000000..6c2d63b5 --- /dev/null +++ b/federated-catalog/fc-01-embedded/fc-connector/build.gradle.kts @@ -0,0 +1,47 @@ +/* + * Copyright (c) 2024 Fraunhofer-Gesellschaft + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Fraunhofer-Gesellschaft - initial API and implementation + * + */ + +plugins { + `java-library` + id("application") + alias(libs.plugins.shadow) +} + +dependencies { + runtimeOnly(project(":federated-catalog:fc-00-basic:federated-catalog-base")) + runtimeOnly(project(":federated-catalog:fc-00-basic:fixed-node-resolver")) + + implementation(libs.edc.connector.core) + implementation(libs.edc.control.plane.core) + implementation(libs.edc.configuration.filesystem) + implementation(libs.edc.management.api) + implementation(libs.edc.dsp) + implementation(libs.edc.iam.mock) + implementation(libs.edc.http) + implementation(libs.edc.edr.store.core) + +} + +application { + mainClass.set("org.eclipse.edc.boot.system.runtime.BaseRuntime") +} + +var distTar = tasks.getByName("distTar") +var distZip = tasks.getByName("distZip") + +tasks.withType { + mergeServiceFiles() + archiveFileName.set("fc-connector.jar") + dependsOn(distTar, distZip) +} diff --git a/federated-catalog/fc-01-embedded/fc-connector/config.properties b/federated-catalog/fc-01-embedded/fc-connector/config.properties new file mode 100644 index 00000000..8df2f80c --- /dev/null +++ b/federated-catalog/fc-01-embedded/fc-connector/config.properties @@ -0,0 +1,33 @@ +edc.participant.id=fc-connector +edc.dsp.callback.address=http://localhost:19194/protocol +web.http.port=29191 +web.http.path=/api +web.http.management.port=29193 +web.http.management.path=/management +web.http.protocol.port=29194 +web.http.protocol.path=/protocol +edc.transfer.proxy.token.signer.privatekey.alias=private-key +edc.transfer.proxy.token.verifier.publickey.alias=public-key +web.http.public.port=29291 +web.http.public.path=/public +web.http.control.port=29192 +web.http.control.path=/control +edc.dataplane.api.public.baseurl=http://localhost:29291/public + + +edc.api.auth.key=password +edc.ids.id=urn:connector:provider +edc.mock.region=us + + +web.http.catalog.path=/api/catalog +web.http.catalog.port=29195 + +edc.catalog.cache.execution.delay.seconds=5 +edc.catalog.cache.execution.period.seconds=5 +edc.catalog.cache.partition.num.crawlers=1 + + + + + diff --git a/federated-catalog/fc-01-embedded/resources/empty-query.json b/federated-catalog/fc-01-embedded/resources/empty-query.json new file mode 100644 index 00000000..a03a6a87 --- /dev/null +++ b/federated-catalog/fc-01-embedded/resources/empty-query.json @@ -0,0 +1,6 @@ +{ + "@context": { + "@vocab": "https://w3id.org/edc/v0.0.1/ns/" + }, + "@type": "QuerySpec" +} \ No newline at end of file diff --git a/federated-catalog/fc-02-standalone/README.md b/federated-catalog/fc-02-standalone/README.md new file mode 100644 index 00000000..0e141bfc --- /dev/null +++ b/federated-catalog/fc-02-standalone/README.md @@ -0,0 +1,107 @@ +# Standalone Federated Catalog + + +In this sample we focus on the implementation of a standalone federated catalog. +Similar to the previous sample [fc-01-embedded](../fc-01-embedded), +the [standalone-fc](./standalone-fc) also builds on the +functionalities of the federated catalog. However, unlike the previous one, it does not +include the additional features of a connector. + +This sample will go through: + +* Implementation of a standalone FC +* Set up of the standalone FC +* Test catalog API endpoint of the standalone FC + + +### 1. Implementation a standalone FC +The [build.gradle.kts](../fc-02-standalone/standalone-fc/build.gradle.kts) +file located in the [fc-02-standalone/standalone-fc](./standalone-fc) +directory includes all the necessary dependencies for creating a standalone federated catalog. This includes the `fc-00-basic:federated-catalog-base`, +and `fc-00-basic:federated-catalog-base` to enable the Catalog Node Resolver. + +```kotlin +dependencies { + runtimeOnly(project(":federated-catalog:fc-00-basic:federated-catalog-base")) + runtimeOnly(project(":federated-catalog:fc-00-basic:static-node-resolver")) +} +``` + +Since we are using `fc-00-basic:federated-catalog-base` as our Catalog Node Resolver here +as well, it will return only +a single target catalog node; the `participant-connector` that we had set up in [fc-00-basic](../fc-00-basic/README.md). +Querying the catalog API will therefore yield just one catalog, which is the contract offered by this connector. + +The [config.properties](./standalone-fc/config.properties) file contains the necessary configurations, +like the `web.http.catalog.path`, which is the catalog API endpoint of this standalone FC. + +```properties +web.http.catalog.path=/api/catalog +web.http.catalog.port=39195 +``` + +### 2. Start the fc-connector +#### Build the fc-connector JAR +Execute this command in project root to build the `standalone-fc` JAR file: + +```bash +./gradlew federated-catalog:fc-02-standalone:standalone-fc:build +``` + + +#### Run the fc-connector + +To run the federated catalog, execute the following command + +```shell +java -Dedc.fs.config=federated-catalog/fc-02-standalone/standalone-fc/config.properties -jar federated-catalog/fc-02-standalone/standalone-fc/build/libs/standalone-fc.jar +``` + +If the execution is successful, then the Catalog API of our standalone FC will listen on port `39195`. + + + +## Test catalog query API +Before requesting the catalog API, make sure the `partcipant-connector` that we have set up in the +[fc-00-basic](../fc-00-basic) is running, and it has a contract offer. + +To get the combined set of catalogs, use the following request: + +```http request +curl -d @federated-catalog/fc-01-embedded/resources/empty-query.json \ + -H 'content-type: application/json' http://localhost:39195/api/catalog/v1alpha/catalog/query \ + -s | jq +``` + +Sample output: +```json +[ + { + "@id": "a8a8cd64-269d-485c-8857-74d08b13ae3c", + "@type": "dcat:Catalog", + "dcat:dataset": { + "@id": "assetId", + "@type": "dcat:Dataset", + "odrl:hasPolicy": { + "@id": "MQ==:YXNzZXRJZA==:MjJmNDlhYTAtM2I3YS00ODkzLTkwZDctNTU5MTZhNmViOWJk" + + }, + "dcat:distribution": [ + + ], + "name": "product description", + "id": "assetId", + "contenttype": "application/json" + }, + "dcat:distribution": [], + "dcat:service": { + + }, + "dspace:participantId": "provider", + "originator": "http://localhost:19194/protocol", + "@context": { + + } + } +] +``` \ No newline at end of file diff --git a/federated-catalog/fc-02-standalone/standalone-fc/build.gradle.kts b/federated-catalog/fc-02-standalone/standalone-fc/build.gradle.kts new file mode 100644 index 00000000..5a585fa5 --- /dev/null +++ b/federated-catalog/fc-02-standalone/standalone-fc/build.gradle.kts @@ -0,0 +1,47 @@ +/* + * Copyright (c) 2024 Fraunhofer-Gesellschaft + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Fraunhofer-Gesellschaft - initial API and implementation + * + */ + +plugins { + `java-library` + id("application") + alias(libs.plugins.shadow) +} + +dependencies { + runtimeOnly(project(":federated-catalog:fc-00-basic:federated-catalog-base")) + runtimeOnly(project(":federated-catalog:fc-00-basic:fixed-node-resolver")) + + implementation(libs.edc.connector.core) + runtimeOnly(libs.edc.boot) + runtimeOnly(libs.edc.control.plane.core) + implementation(libs.edc.configuration.filesystem) + runtimeOnly(libs.edc.token.core) + implementation(libs.edc.http) + runtimeOnly(libs.edc.dsp) + implementation(libs.edc.iam.mock) + +} + +application { + mainClass.set("org.eclipse.edc.boot.system.runtime.BaseRuntime") +} + +var distTar = tasks.getByName("distTar") +var distZip = tasks.getByName("distZip") + +tasks.withType { + mergeServiceFiles() + archiveFileName.set("standalone-fc.jar") + dependsOn(distTar, distZip) +} diff --git a/federated-catalog/fc-02-standalone/standalone-fc/config.properties b/federated-catalog/fc-02-standalone/standalone-fc/config.properties new file mode 100644 index 00000000..72c24270 --- /dev/null +++ b/federated-catalog/fc-02-standalone/standalone-fc/config.properties @@ -0,0 +1,14 @@ +web.http.port=39191 +web.http.path=/api + +web.http.catalog.path=/api/catalog +web.http.catalog.port=39195 + +edc.catalog.cache.execution.delay.seconds=5 +edc.catalog.cache.execution.period.seconds=5 +edc.catalog.cache.partition.num.crawlers=1 + + + + + diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 43bbdf79..64966e8f 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -74,5 +74,11 @@ testcontainers-hashicorp-vault = { module = "org.testcontainers:vault", version. azure-storage-blob = { module = "com.azure:azure-storage-blob", version = "12.29.0" } minio-io = { module = "io.minio:minio", version = "8.5.14" } +# federated catalog modules +edc-fc-spi-core = { module = "org.eclipse.edc:federated-catalog-spi", version.ref = "edc" } +edc-fc-core = { module = "org.eclipse.edc:federated-catalog-core", version.ref = "edc" } +edc-fc-ext-api = { module = "org.eclipse.edc:federated-catalog-api", version.ref = "edc" } +edc-fc-spi-crawler = { module = "org.eclipse.edc:crawler-spi", version.ref = "edc" } + [plugins] shadow = { id = "com.github.johnrengelman.shadow", version = "8.1.1" } diff --git a/settings.gradle.kts b/settings.gradle.kts index 90c0d9cb..88da6243 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -52,6 +52,13 @@ include(":policy:policy-01-policy-enforcement:policy-enforcement-provider") include(":policy:policy-01-policy-enforcement:policy-enforcement-consumer") include(":policy:policy-01-policy-enforcement:policy-functions") +// federated catalog +include(":federated-catalog:fc-00-basic:federated-catalog-base") +include(":federated-catalog:fc-00-basic:fixed-node-resolver") +include(":federated-catalog:fc-01-embedded:fc-connector") +include(":federated-catalog:fc-02-standalone:standalone-fc") + + // modules for code samples ------------------------------------------------------------------------ include(":advanced:advanced-02-custom-runtime") diff --git a/system-tests/build.gradle.kts b/system-tests/build.gradle.kts index 03370623..11855353 100644 --- a/system-tests/build.gradle.kts +++ b/system-tests/build.gradle.kts @@ -9,6 +9,7 @@ * * Contributors: * Bayerische Motoren Werke Aktiengesellschaft (BMW AG) - initial API and implementation + * Fraunhofer-Gesellschaft - dependencies for Federated Catalog Tests * */ @@ -55,6 +56,10 @@ dependencies { testCompileOnly(project(":transfer:transfer-05-file-transfer-cloud:cloud-transfer-provider")) testCompileOnly(project(":transfer:transfer-05-file-transfer-cloud:cloud-transfer-consumer")) testCompileOnly(project(":transfer:transfer-05-file-transfer-cloud:transfer-file-cloud")) + + testCompileOnly(project(":federated-catalog:fc-00-basic:fixed-node-resolver")) + testCompileOnly(project(":federated-catalog:fc-01-embedded:fc-connector")) + testCompileOnly(project(":federated-catalog:fc-02-standalone:standalone-fc")) } tasks.compileJava { diff --git a/system-tests/src/test/java/org/eclipse/edc/samples/common/FederatedCatalogCommon.java b/system-tests/src/test/java/org/eclipse/edc/samples/common/FederatedCatalogCommon.java new file mode 100644 index 00000000..1705f5dc --- /dev/null +++ b/system-tests/src/test/java/org/eclipse/edc/samples/common/FederatedCatalogCommon.java @@ -0,0 +1,116 @@ +/* + * Copyright (c) 2024 Fraunhofer-Gesellschaft + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Fraunhofer-Gesellschaft - initial API and implementation + * + */ + +package org.eclipse.edc.samples.common; + +import com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.HttpStatus; +import io.restassured.http.ContentType; +import org.eclipse.edc.junit.extensions.EmbeddedRuntime; +import org.eclipse.edc.junit.extensions.RuntimeExtension; +import org.eclipse.edc.junit.extensions.RuntimePerClassExtension; + +import java.util.Map; + +import static io.restassured.RestAssured.given; +import static org.eclipse.edc.samples.common.FileTransferCommon.getFileContentFromRelativePath; +import static org.eclipse.edc.samples.common.FileTransferCommon.getFileFromRelativePath; +import static org.eclipse.edc.samples.common.PrerequisitesCommon.API_KEY_HEADER_KEY; +import static org.eclipse.edc.samples.common.PrerequisitesCommon.API_KEY_HEADER_VALUE; +import static org.eclipse.edc.samples.util.TransferUtil.post; +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.CoreMatchers.not; +import static org.hamcrest.Matchers.emptyString; + +public class FederatedCatalogCommon { + private static final String CREATE_ASSET_FILE_PATH = "transfer/transfer-01-negotiation/resources/create-asset.json"; + private static final String V3_ASSETS_PATH = "/v3/assets"; + private static final String ASSET_ID = "@id"; + + private static final String STANDALONE_FC_MODULE_PATH = ":federated-catalog:fc-02-standalone:standalone-fc"; + private static final String FC_CONNECTOR_MODULE_PATH = ":federated-catalog:fc-01-embedded:fc-connector"; + private static final String STANDALONE_FC = "standalone-fc"; + private static final String EMBEDDED_FC = "fc-connector"; + + private static final String EDC_KEYSTORE = "edc.keystore"; + private static final String EDC_KEYSTORE_PASSWORD = "edc.keystore.password"; + private static final String EDC_FS_CONFIG = "edc.fs.config"; + private static final String CERT_PFX_FILE_PATH = "transfer/transfer-00-prerequisites/resources/certs/cert.pfx"; + private static final String KEYSTORE_PASSWORD = "123456"; + + private static final String STANDALONE_FC_CONFIG_PROPERTIES_FILE_PATH = "federated-catalog/fc-02-standalone/standalone-fc/config.properties"; + private static final String FC_CONNECTOR_CONFIG_PROPERTIES_FILE_PATH = "federated-catalog/fc-01-embedded/fc-connector/config.properties"; + + private static final String CRAWLER_EXECUTION_DELAY = "edc.catalog.cache.execution.delay.seconds"; + public static final int CRAWLER_EXECUTION_DELAY_VALUE = 5; + private static final String CRAWLER_EXECUTION_PERIOD = "edc.catalog.cache.execution.period.seconds"; + public static final int CRAWLER_EXECUTION_PERIOD_VALUE = 5; + public static final int TIMEOUT = 5 * CRAWLER_EXECUTION_PERIOD_VALUE; + + public static final String EMBEDDED_FC_CATALOG_API_ENDPOINT = "http://localhost:29195/api/catalog/v1alpha/catalog/query"; + public static final String STANDALONE_FC_CATALOG_API_ENDPOINT = "http://localhost:39195/api/catalog/v1alpha/catalog/query"; + public static final String EMPTY_QUERY_FILE_PATH = "federated-catalog/fc-01-embedded/resources/empty-query.json"; + public static final String TYPE = "[0].@type"; + public static final String CATALOG = "dcat:Catalog"; + public static final String DATASET_ASSET_ID = "[0].'dcat:dataset'.@id"; + + public static RuntimeExtension getFcEmbeddedConnector() { + return getRuntime(FC_CONNECTOR_MODULE_PATH, EMBEDDED_FC, FC_CONNECTOR_CONFIG_PROPERTIES_FILE_PATH); + } + + public static RuntimeExtension getStandaloneFc() { + return getRuntime(STANDALONE_FC_MODULE_PATH, STANDALONE_FC, STANDALONE_FC_CONFIG_PROPERTIES_FILE_PATH); + } + + private static RuntimeExtension getRuntime( + String modulePath, + String moduleName, + String configPropertiesFilePath + ) { + return new RuntimePerClassExtension(new EmbeddedRuntime( + moduleName, + Map.of( + EDC_KEYSTORE, getFileFromRelativePath(CERT_PFX_FILE_PATH).getAbsolutePath(), + EDC_KEYSTORE_PASSWORD, KEYSTORE_PASSWORD, + EDC_FS_CONFIG, getFileFromRelativePath(configPropertiesFilePath).getAbsolutePath(), + CRAWLER_EXECUTION_DELAY, Integer.toString(CRAWLER_EXECUTION_DELAY_VALUE), + CRAWLER_EXECUTION_PERIOD, Integer.toString(CRAWLER_EXECUTION_PERIOD_VALUE) + ), + modulePath + )); + } + + public static String createAsset() { + return post(PrerequisitesCommon.PROVIDER_MANAGEMENT_URL + V3_ASSETS_PATH, + getFileContentFromRelativePath(CREATE_ASSET_FILE_PATH), + ASSET_ID); + } + + public static String postAndAssertType(String url, String requestBody, String jsonPath) { + return given() + .headers(API_KEY_HEADER_KEY, API_KEY_HEADER_VALUE) + .contentType(ContentType.JSON) + .body(requestBody) + .when() + .post(url) + .then() + .log().ifError() + .statusCode(HttpStatus.SC_OK) + .body(TYPE, not(emptyString())) + .body(TYPE, is(CATALOG)) + .extract() + .jsonPath() + .get(jsonPath); + } + +} diff --git a/system-tests/src/test/java/org/eclipse/edc/samples/federated/catalog/FederatedCatalog01embeddedTest.java b/system-tests/src/test/java/org/eclipse/edc/samples/federated/catalog/FederatedCatalog01embeddedTest.java new file mode 100644 index 00000000..b40e362d --- /dev/null +++ b/system-tests/src/test/java/org/eclipse/edc/samples/federated/catalog/FederatedCatalog01embeddedTest.java @@ -0,0 +1,71 @@ +/* + * Copyright (c) 2024 Fraunhofer-Gesellschaft + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Fraunhofer-Gesellschaft - initial API and implementation + * + */ + +package org.eclipse.edc.samples.federated.catalog; + +import org.eclipse.edc.junit.annotations.EndToEndTest; +import org.eclipse.edc.junit.extensions.RuntimeExtension; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; + +import java.time.Clock; +import java.time.Duration; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.awaitility.Awaitility.await; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.CRAWLER_EXECUTION_DELAY_VALUE; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.CRAWLER_EXECUTION_PERIOD_VALUE; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.DATASET_ASSET_ID; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.EMBEDDED_FC_CATALOG_API_ENDPOINT; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.EMPTY_QUERY_FILE_PATH; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.TIMEOUT; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.createAsset; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.getFcEmbeddedConnector; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.postAndAssertType; +import static org.eclipse.edc.samples.common.FileTransferCommon.getFileContentFromRelativePath; +import static org.eclipse.edc.samples.common.NegotiationCommon.createContractDefinition; +import static org.eclipse.edc.samples.common.NegotiationCommon.createPolicy; +import static org.eclipse.edc.samples.common.PrerequisitesCommon.getProvider; + +@EndToEndTest +public class FederatedCatalog01embeddedTest { + + @RegisterExtension + static final RuntimeExtension PARTICIPANT_CONNECTOR = getProvider(); + + @RegisterExtension + static final RuntimeExtension FC_CONNECTOR = getFcEmbeddedConnector(); + + @Test + void shouldStartConnector() { + assertThat(PARTICIPANT_CONNECTOR.getService(Clock.class)).isNotNull(); + assertThat(FC_CONNECTOR.getService(Clock.class)).isNotNull(); + } + + @Test + void runSampleSteps() { + String assetId = createAsset(); + createPolicy(); + createContractDefinition(); + + await() + .atMost(Duration.ofSeconds(TIMEOUT)) + .pollDelay(Duration.ofSeconds(CRAWLER_EXECUTION_DELAY_VALUE)) + .pollInterval(Duration.ofSeconds(CRAWLER_EXECUTION_PERIOD_VALUE)) + .ignoreExceptions() + .until(() -> postAndAssertType(EMBEDDED_FC_CATALOG_API_ENDPOINT, getFileContentFromRelativePath(EMPTY_QUERY_FILE_PATH), DATASET_ASSET_ID), + id -> id.equals(assetId)); + } + +} diff --git a/system-tests/src/test/java/org/eclipse/edc/samples/federated/catalog/FederatedCatalog02standaloneTest.java b/system-tests/src/test/java/org/eclipse/edc/samples/federated/catalog/FederatedCatalog02standaloneTest.java new file mode 100644 index 00000000..d748d959 --- /dev/null +++ b/system-tests/src/test/java/org/eclipse/edc/samples/federated/catalog/FederatedCatalog02standaloneTest.java @@ -0,0 +1,70 @@ +/* + * Copyright (c) 2024 Fraunhofer-Gesellschaft + * + * This program and the accompanying materials are made available under the + * terms of the Apache License, Version 2.0 which is available at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * SPDX-License-Identifier: Apache-2.0 + * + * Contributors: + * Fraunhofer-Gesellschaft - initial API and implementation + * + */ + +package org.eclipse.edc.samples.federated.catalog; + +import org.eclipse.edc.junit.annotations.EndToEndTest; +import org.eclipse.edc.junit.extensions.RuntimeExtension; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.RegisterExtension; + +import java.time.Clock; +import java.time.Duration; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.awaitility.Awaitility.await; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.CRAWLER_EXECUTION_DELAY_VALUE; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.CRAWLER_EXECUTION_PERIOD_VALUE; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.DATASET_ASSET_ID; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.EMPTY_QUERY_FILE_PATH; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.STANDALONE_FC_CATALOG_API_ENDPOINT; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.TIMEOUT; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.createAsset; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.getStandaloneFc; +import static org.eclipse.edc.samples.common.FederatedCatalogCommon.postAndAssertType; +import static org.eclipse.edc.samples.common.FileTransferCommon.getFileContentFromRelativePath; +import static org.eclipse.edc.samples.common.NegotiationCommon.createContractDefinition; +import static org.eclipse.edc.samples.common.NegotiationCommon.createPolicy; +import static org.eclipse.edc.samples.common.PrerequisitesCommon.getProvider; + +@EndToEndTest +public class FederatedCatalog02standaloneTest { + @RegisterExtension + static final RuntimeExtension PARTICIPANT_CONNECTOR = getProvider(); + + @RegisterExtension + static final RuntimeExtension STANDALONE_FC_RUNTIME = getStandaloneFc(); + + @Test + void shouldStartRuntimes() { + assertThat(PARTICIPANT_CONNECTOR.getService(Clock.class)).isNotNull(); + assertThat(STANDALONE_FC_RUNTIME.getService(Clock.class)).isNotNull(); + } + + @Test + void runSampleSteps() { + String assetId = createAsset(); + createPolicy(); + createContractDefinition(); + + await() + .atMost(Duration.ofSeconds(TIMEOUT)) + .pollDelay(Duration.ofSeconds(CRAWLER_EXECUTION_DELAY_VALUE)) + .pollInterval(Duration.ofSeconds(CRAWLER_EXECUTION_PERIOD_VALUE)) + .ignoreExceptions() + .until(() -> postAndAssertType(STANDALONE_FC_CATALOG_API_ENDPOINT, getFileContentFromRelativePath(EMPTY_QUERY_FILE_PATH), DATASET_ASSET_ID), + id -> id.equals(assetId)); + } + +}