feat: add federated catalog samples for an embedded and a standalone federated catalog

federated-catalog/README.md

@@ -0,0 +1,40 @@
+# Federated Catalog Samples
+
+The samples in this section focus on the topic of Federated Catalogs.
+
+The Federated Catalog (FC) functions as an aggregated repository of
+catalogs obtained from multiple
+participants in the dataspace. To accomplish this, the FC utilizes crawlers
+that periodically crawl the catalogs from each participant and store this list
+of catalogs in a local cache.
+By maintaining this locally cached version of catalogs, it eliminates the need to query
+each participant individually, resulting in faster and more reliable queries.
+
+
+The following samples shows how to
+* implement, build and run different versions of FC e.g.
+    * standalone,
+    * embedded.
+
+
+## Samples
+
+### [FC sample 00](./fc-00-basic/README.md): Federated Catalog Prerequisites
+The purpose of this example is to make preparations for implementing Federated Catalog (FC).
+We'll set up a basic federated catalog that includes necessary dependencies for triggering the FC,
+along with some other modules to demonstrate FC functionalities.
+
+---
+
+### Implement Different Versions of FC
+### [FC sample 01](./fc-01-embedded/README.md): Implement an embedded federated catalog 
+This sample demonstrates how we can implement a federated catalog which is embedded in a connector.
+The connector exposes a catalog endpoint that serves the consolidated list of catalogs.
+### [FC sample 02](./fc-02-standalone/README.md): Implement a standalone federated catalog
+
+In this sample we focus on the implementation of
+a standalone federated catalog. Unlike the previous sample,
+a standalone federated catalog will not have the added functionalities of a connector. However, it also
+exposes a catalog API that serves the list of catalogs.
+
+---

federated-catalog/fc-00-basic/README.md

@@ -0,0 +1,97 @@
+# Federated Catalog Prerequisites
+
+The purpose of this example is to make preparations for implementing Federated Catalog (FC) 
+and set up additional requirements for testing the FC functionalities.
+For this purpose, we will be covering the following in this scope.
+* `federated-catalog-base`: A basic federated catalog that includes necessary dependencies for triggering the FC.
+* `participant-connector`: A connector with a contract offer. We will require this for testing the functionalities of the FCs in our later samples.
+* `fixed-node-resolver`: A mock node directory resolver. 
+It provides a fixed Target Catalog Node, which represents the `participant-connector`.
+
+
+
+### federated-catalog-base
+The [federated-catalog-base](../fc-00-basic/federated-catalog-base) will be used as a foundational module in our upcoming samples to trigger the FC.
+It provides a [build.gradle.kts](./federated-catalog-base/build.gradle.kts) file that includes only the dependencies
+essential for FC, without any additional functionality.
+```shell
+...
+dependencies {
+    implementation(libs.edc.fc.spi.crawler)
+    runtimeOnly(libs.fc.core)
+    runtimeOnly(libs.fc.ext.api)
+}
+...
+```
+Any further dependencies will be added in the later samples based on their use cases.
+
+
+### fixed-node-resolver
+The Federated Catalog requires a list of Target Catalog Nodes, which are essentially the DSP endpoints of the dataspace participants.
+The catalog crawler then crawls these listed endpoints to collect their offered catalogs. 
+This list of Target Nodes is resolved by a Catalog Node Resolver which implements the [TargetNodeDirectory](https://github.com/eclipse-edc/FederatedCatalog/blob/main/spi/crawler-spi/src/main/java/org/eclipse/edc/crawler/spi/TargetNodeDirectory.java).
+Check out [eclipse-edc/FederatedCatalog](https://github.com/eclipse-edc/FederatedCatalog/tree/main) for further information on this topic.
+
+
+In this module, we've included a fixed Node Resolver, [fixed-node-resolver](./fixed-node-resolver)
+that simply returns a hard-coded Target Node of the `participant-connector`.
+However, we will not cover the implementation of the resolver in this sample; that will be explained in detail in later samples.
+
+
+The purpose of including this [`fixed-node-resolver`](./fixed-node-resolver)
+as a prerequisite, is the fact that we need to have some form of Target Node Resolver to demonstrate the functionality
+of the federated catalogs that we are going to build in sample 
+[fc-01-embedded](../fc-01-embedded) and [fc-02-standalone](../fc-02-standalone).
+
+### participant connector
+
+When the federated catalog boots up, the crawler begins periodically invoking the Target Nodes returned by the 
+Node Resolver and collecting the catalogs offered by these nodes. To test whether our federated catalogs 
+(which we will build in later samples: [fc-01-embedded](../fc-01-embedded) and [fc-02-standalone](../fc-02-standalone)) can successfully request and retrieve these catalogs, we need at least one connector with a contract offer.
+
+Therefore, in this section, we will start a connector and then create a contract 
+for this connector. In the future samples, we will refer to it as `participant-connector`.
+This `participant-connector` will function as a provider.
+We will use the resources from the [transfer](../../transfer) sample to set up this connector. In the rest of this section we will,
+* run the `participant-connector`
+* create an asset for this `participant-connector`
+* create a policy
+* create a contract offer
+
+Although these topics were covered in the [transfer](../../transfer) section, we’ll document all the necessary commands here for easier execution.
+
+
+#### Build connector jar
+Use the following command to build a connector jar.
+```shell
+./gradlew transfer:transfer-00-prerequisites:connector:build
+```
+#### Run the connector
+Execute the following to run the connector jar.
+```shell
+java -Dedc.keystore=transfer/transfer-00-prerequisites/resources/certs/cert.pfx -Dedc.keystore.password=123456 -Dedc.fs.config=transfer/transfer-00-prerequisites/resources/configuration/provider-configuration.properties -jar transfer/transfer-00-prerequisites/connector/build/libs/connector.jar
+```
+
+---
+
+Once the connector is running, carry out the commands below in sequence.
+#### Create an asset
+```shell
+curl -d @transfer/transfer-01-negotiation/resources/create-asset.json \
+  -H 'content-type: application/json' http://localhost:19193/management/v3/assets \
+  -s | jq
+```
+
+#### Create a policy
+```bash
+curl -d @transfer/transfer-01-negotiation/resources/create-policy.json \
+  -H 'content-type: application/json' http://localhost:19193/management/v3/policydefinitions \
+  -s | jq
+```
+
+#### Create a contract definition
+```bash
+curl -d @transfer/transfer-01-negotiation/resources/create-contract-definition.json \
+  -H 'content-type: application/json' http://localhost:19193/management/v3/contractdefinitions \
+  -s | jq
+```

federated-catalog/fc-00-basic/federated-catalog-base/build.gradle.kts

@@ -0,0 +1,26 @@
+/*
+ *  Copyright (c) 2024 Fraunhofer-Gesellschaft
+ *
+ *  This program and the accompanying materials are made available under the
+ *  terms of the Apache License, Version 2.0 which is available at
+ *  https://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  SPDX-License-Identifier: Apache-2.0
+ *
+ *  Contributors:
+ *       Fraunhofer-Gesellschaft - initial API and implementation
+ *
+ */
+
+plugins {
+    `java-library`
+    id("application")
+}
+
+dependencies {
+    implementation(libs.edc.fc.spi.crawler)
+    runtimeOnly(libs.edc.fc.core)
+    runtimeOnly(libs.edc.fc.ext.api)
+}
+
+

federated-catalog/fc-00-basic/fixed-node-resolver/build.gradle.kts

@@ -0,0 +1,23 @@
+/*
+ *  Copyright (c) 2024 Fraunhofer-Gesellschaft
+ *
+ *  This program and the accompanying materials are made available under the
+ *  terms of the Apache License, Version 2.0 which is available at
+ *  https://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  SPDX-License-Identifier: Apache-2.0
+ *
+ *  Contributors:
+ *       Fraunhofer-Gesellschaft - initial API and implementation
+ *
+ */
+
+plugins {
+    `java-library`
+    id("application")
+}
+
+dependencies {
+    implementation(libs.edc.fc.spi.crawler)
+}
+

federated-catalog/fc-00-basic/fixed-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectory.java

@@ -0,0 +1,41 @@
+/*
+ *  Copyright (c) 2024 Fraunhofer-Gesellschaft
+ *
+ *  This program and the accompanying materials are made available under the
+ *  terms of the Apache License, Version 2.0 which is available at
+ *  https://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  SPDX-License-Identifier: Apache-2.0
+ *
+ *  Contributors:
+ *       Fraunhofer-Gesellschaft - initial API and implementation
+ *
+ */
+
+package org.eclipse.edc.sample.extension.fc;
+
+import org.eclipse.edc.crawler.spi.TargetNode;
+import org.eclipse.edc.crawler.spi.TargetNodeDirectory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class CatalogNodeDirectory implements TargetNodeDirectory {
+
+    @Override
+    public List<TargetNode> getAll() {
+        List<String> protocolList = new ArrayList<>();
+        protocolList.add("dataspace-protocol-http");
+
+        TargetNode participantNode = new TargetNode("https://w3id.org/edc/v0.0.1/ns/",
+                "provider",
+                "http://localhost:19194/protocol", protocolList);
+
+        return List.of(participantNode);
+    }
+
+    @Override
+    public void insert(TargetNode targetNode) {
+
+    }
+}

federated-catalog/fc-00-basic/fixed-node-resolver/src/main/java/org/eclipse/edc/sample/extension/fc/CatalogNodeDirectoryExtension.java

@@ -0,0 +1,28 @@
+/*
+ *  Copyright (c) 2024 Fraunhofer-Gesellschaft
+ *
+ *  This program and the accompanying materials are made available under the
+ *  terms of the Apache License, Version 2.0 which is available at
+ *  https://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  SPDX-License-Identifier: Apache-2.0
+ *
+ *  Contributors:
+ *       Fraunhofer-Gesellschaft - initial API and implementation
+ *
+ */
+
+package org.eclipse.edc.sample.extension.fc;
+
+import org.eclipse.edc.crawler.spi.TargetNodeDirectory;
+import org.eclipse.edc.runtime.metamodel.annotation.Provider;
+import org.eclipse.edc.spi.system.ServiceExtension;
+
+public class CatalogNodeDirectoryExtension implements ServiceExtension {
+
+    @Provider 
+    public TargetNodeDirectory federatedCacheNodeDirectory() {
+        return new CatalogNodeDirectory();
+    }
+
+}

federated-catalog/fc-00-basic/fixed-node-resolver/src/main/resources/META-INF/services/org.eclipse.edc.spi.system.ServiceExtension

@@ -0,0 +1 @@
+org.eclipse.edc.sample.extension.fc.CatalogNodeDirectoryExtension

federated-catalog/fc-01-embedded/README.md

@@ -0,0 +1,123 @@
+# Embedded Federated Catalog
+
+
+This sample demonstrates how we can implement a federated catalog which is embedded in a connector.
+We will build one connector of such type and will call it `fc-connector`.
+As discussed in the prerequisite sample [fc-00-basic](../fc-00-basic/README.md), 
+we will be using the Node Resolver 
+[fixed-node-resolver](../fc-00-basic/fixed-node-resolver) as a dependency in our embedded federated catalog. 
+Also, the `participant-connector` we set up in the previous sample should still be running. 
+This `participant-connector` will act as a provider, while the new `fc-connector` will act as a consumer.
+
+
+
+This sample will go through:
+
+* Implementation of an embedded FC
+* Set up of the embedded FC, `fc-connector`
+* Test catalog API endpoint of the `fc-connector`
+
+
+### 1. Implementation the fc-connector
+The [build.gradle.kts](../fc-01-embedded/fc-connector/build.gradle.kts) 
+file located in the [fc-01-embedded/fc-connector](../fc-01-embedded/fc-connector) 
+directory includes all the necessary dependencies for creating a connector, along with the `fc-00-basic:federated-catalog-base` 
+needed to trigger the FC. Additionally, we need to add `fc-00-basic:federated-catalog-base` as a dependency to enable the Catalog Node Resolver.
+
+```kotlin
+dependencies {
+    runtimeOnly(project(":federated-catalog:fc-00-basic:federated-catalog-base"))
+    runtimeOnly(project(":federated-catalog:fc-00-basic:static-node-resolver"))
+}
+```
+
+The [config.properties](../fc-01-embedded/fc-connector/config.properties) 
+file contains the necessary configurations 
+for this `fc-connector`, including the standard settings for a regular connector, along with additional configurations for a 
+federated catalog, such as catalog api endpoint and crawler execution interval.
+
+```properties
+web.http.catalog.path=/api/catalog
+web.http.catalog.port=29195
+
+edc.catalog.cache.execution.delay.seconds=5
+edc.catalog.cache.execution.period.seconds=5
+edc.catalog.cache.partition.num.crawlers=5
+```
+
+### 2. Start the fc-connector
+#### Build the fc-connector JAR
+Execute this command in project root to build the `fc-connector` JAR file:
+
+```bash
+./gradlew federated-catalog:fc-01-embedded:fc-connector:build
+```
+
+
+#### Run the fc-connector
+
+To run the connector, execute the following command
+
+```shell
+java -Dedc.fs.config=federated-catalog/fc-01-embedded/fc-connector/config.properties -jar federated-catalog/fc-01-embedded/fc-connector/build/libs/fc-connector.jar
+```
+
+If the execution is successful, then the Catalog API of our `fc-connector` will listen on port `29195`.
+
+If you observe the logs, you can see the following recurring lines,
+
+> DEBUG 2024-11-14T13:53:48.472700883 [ExecutionManager] Run pre-execution task
+>
+>DEBUG 2024-11-14T13:53:48.494149928 [ExecutionManager] Loaded 1 work items from storage
+>
+>DEBUG 2024-11-14T13:53:48.495574504 [ExecutionManager] Crawler parallelism is 1, based on config and number of work items
+>
+>DEBUG 2024-11-14T13:53:48.497891576 [ExecutionManager] Crawler-f81f5514-5c7f-44aa-94bb-16998861789b: WorkItem acquired
+>
+>DEBUG 2024-11-14T13:53:48.790873233 [ExecutionManager] Crawler [Crawler-f81f5514-5c7f-44aa-94bb-16998861789b] is done
+
+
+This means our FC crawler is running, and the crawler found one node, which is the `participant-connector` we had set up before.
+
+
+
+### 3. Test catalog query API
+
+To query the catalogs from `fc-connector` side, we can now call the catalog API of our embedded federated catalog. 
+Use the following request to invoke the catalog API:
+
+```http request
+curl -d @federated-catalog/fc-01-embedded/resources/empty-query.json \
+  -H 'content-type: application/json' http://localhost:29195/api/catalog/v1alpha/catalog/query \
+  -s | jq
+```
+
+Sample output:
+```json
+[
+  {
+    "@id": "a8a8cd64-269d-485c-8857-74d08b13ae3c",
+    "@type": "dcat:Catalog",
+    "dcat:dataset": {
+      "@id": "assetId",
+      "@type": "dcat:Dataset",
+      "odrl:hasPolicy": {
+        "@id": "MQ==:YXNzZXRJZA==:MjJmNDlhYTAtM2I3YS00ODkzLTkwZDctNTU5MTZhNmViOWJk"
+      },
+      "dcat:distribution": [
+      ],
+      "name": "product description",
+      "id": "assetId",
+      "contenttype": "application/json"
+    },
+    "dcat:distribution": [],
+    "dcat:service": {
+
+    },
+    "dspace:participantId": "provider",
+    "originator": "http://localhost:19194/protocol",
+    "@context": {
+    }
+  }
+]
+```