45 add readingwriting pki files from $pki config dir (#46)

* Add PKI config dir logic and restructure DSH module

* Update README and CHANGELOG

* cargo fmt

* improve some internal functions amd a[i doc

* restore example

* update readme

* cargo fmt

* Add unit test

* update readme

* update connect readme

* update readme

* cargo fmt

* cargo clippy suggestions

---------

Co-authored-by: Frank Hol <frank.hol@kpn.com>

README.md

@@ -1,5 +1,5 @@
 # Dsh-sdk-platform-rs
-This repository contains the Rust SDK for the Data Sharing Hub (DSH) platform. 
+This repository contains the Rust SDK for the Data Sharing Hub (DSH) platform and some related utility crates.
 
 ## Dsh_sdk
 The [dsh_sdk](dsh_sdk) is a Rust library that provides a simple interface to interact with the DSH platform. The SDK is used to create and manage data streams, and to send data to the DSH platform.
@@ -14,7 +14,7 @@ The [docker](docker) directory contains a docker-compose file that can be used f
 ---
 
 ### Changelog
-See [CHANGELOG.md](CHANGELOG.md) for all changes per version.
+See DSH_SDK [CHANGELOG.md](dsh_sdk/CHANGELOG.md) for all changes per version for SDK.
 
 ### Contributing
 See [CONTRIBUTING.md](CONTRIBUTING.md) for more information on how to contribute to this project.

CHANGELOG.md → dsh_sdk/CHANGELOG.md

@@ -5,6 +5,15 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+### Added
+- Add loading (PEM) Certificates and Keys from $PKI_CONFIG_DIR
+
+### Changed
+- Restructure of the private functions to make it more modular
+- Improved logging
+- Improved API Documentation
+
 ## [0.4.0] - 2024-04-25
 
 ### Fixed

dsh_sdk/CONNECT_PROXY_VPN_LOCAL.md

@@ -0,0 +1,39 @@
+# How to run
+The SDK is compatible with running in a container on a DSH tenant, on DSH System Space, on a machine with Kafka
+Proxy/VPN or on a local machine to a local Kafka
+
+- [DSH](#dsh)
+- [System Space](#system-space)
+- [Kafka Proxy/VPN](#kafka-proxyvpn)
+- [Local](#local)
+
+## DSH
+The following environment variables are required to run on DSH, and are set by DSH automatically:
+- `MESOS_TASK_ID` - The task id of the running container
+- `MARATHON_APP_ID` - Includes the tenant name of the running container
+- `DSH_CA_CERTIFICATE` - The CA certificate of DSH
+- `DSH_SECRET_TOKEN` - The secret token to authenticate to DSH
+
+### System Space
+When running on DSH System Space, the following environment variables are required:
+- `DSH_SECRET_TOKEN_PATH` - The path to the secret token file.
+
+## Kafka Proxy/VPN
+When running on a machine with Kafka Proxy/VPN, the following environment variables are required:
+- `PKI_CONFIG_DIR` - The path to the directory containing the certificates and private key
+- `DSH_TENANT_NAME` - The tenant name of which you want to connect to
+- `KAFKA_BOOTSTRAP_SERVERS` - The hostnames of the Kafka brokers
+
+### Note!
+Currently only PEM formatted certificates and keys are supported. Make sure to convert your certificates and key to PEM format if they are not already.
+
+## Local
+You can start the [docker-compose](../docker/docker-compose.yml) file to start a local Kafka broker and Schema Registry.
+
+When no environment variables are set, it will default to a local configuration.
+- Kafka will be set to `localhost:9092` and uses plaintext instead of SSL
+- Schema Registry will be set to `localhost:8081/apis/ccompat/v7`
+
+You can overwrite this by providing a [local_datastreams.json](https://github.com/kpn-dsh/dsh-sdk-platform-rs/blob/main/dsh_sdklocal_datastreams.json) file to the root of the project or by setting the following environment variables.
+- `KAFKA_BOOTSTRAP_SERVERS` - The hostnames of the Kafka brokers
+- `SCHEMA_REGISTRY_HOST` - The host of the Schema Registry

dsh_sdk/Cargo.toml

@@ -21,6 +21,7 @@ hyper = { version = "1.3", features = ["server", "http1"], optional = true }
 hyper-util = { version = "0.1", features = ["tokio"], optional = true }
 lazy_static = { version = "1.4", optional = true }
 log = "0.4"
+pem = "3"
 prometheus = { version = "0.13", features = ["process"], optional = true }
 rcgen = { version = "0.13", optional = true }
 reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "blocking"], optional = true }

dsh_sdk/README.md

@@ -16,8 +16,11 @@ This library can be used to interact with the DSH Platform. It is intended to be
 - Prometheus Metrics (web server and re-export of metrics crate)
 - Dead Letter Queue (experimental)
 
-## Usage
+### Note
+Rdkafka and thereby this library is dependent on CMAKE. Make sure it is installed in your environment and/or Dockerfile where you are compiling.
+See [dockerfile](../example_dsh_service/Dockerfile) for an example.
 
+## Usage
 To use this SDK with the default features in your project, add the following to your Cargo.toml file:
 
 ```toml
@@ -46,8 +49,11 @@ fn main() -> Result<(), Box<dyn std::error::Error>>{
 }
 ```
 
-## Feature flags
+## Connect to DSH
+The SDK is compatible with running in a container on a DSH tenant, on DSH System Space, on a machine with Kafka Proxy/VPN or on a local machine to a local Kafka. 
+See [CONNECT_PROXY_VPN_LOCAL](/dsh_sdk/CONNECT_PROXY_VPN_LOCAL.md) for more info.
 
+## Feature flags
 The following features are available in this library and can be enabled/disabled in your Cargo.toml file.:
 
 | **feature** | **default** | **Description** |
@@ -93,33 +99,22 @@ The default RDKafka config can be overwritten by setting the following environme
 - Required: `false`
 - Options: smallest, earliest, beginning, largest, latest, end
 
+### `SCHEMA_REGISTRY_HOST`
+- Usage: Overwrite Schema Registry host
+- Default: Schema Registry based on datastreams
+- Required: `false`
+
 ## Api doc
 See the [api documentation](https://docs.rs/dsh_sdk/latest/dsh_sdk/) for more information on how to use this library.
 
-### Local development
-You can start the [docker-compose](../docker/docker-compose.yml) file to start a local Kafka broker and Schema Registry.
-
-When running the SDK on your local machine, it will automatically try to connect to the local Kafka broker and Schema Registry
-
-| Service | Host |
-| --- | --- |
-| Kafka | `localhost:9092` |
-| Schema Registry | `localhost:8081/apis/ccompat/v7` |
-
-If you want manipulate these endpoints, or want to use specific datastream info, you can add a [local_datastreams.json](local_datastreams.json) to your project root to overwrite the default values or set the environment variables accordingly.
-
-### Note
-Rdkafka and thereby this library is dependent on CMAKE. Make sure it is installed in your environment and/or Dockerfile where you are compiling.
-See dockerfile in [example_dsh_service](../example_dsh_service/Dockerfile) for an example.
-
 ## Examples
 See folder [dsh_sdk/examples](/examples/) for simple examples on how to use the SDK.
 
 ### Full service example
-See folder [example_dsh_service](../example_dsh_service/) for a full service, including how to build the Rust project and post it to Harbor. See [readme](/example_dsh_service/README.md) for more information.
+See folder [example_dsh_service](../example_dsh_service/) for a full service, including how to build the Rust project and post it to Harbor. See [readme](../example_dsh_service/README.md) for more information.
 
 ## Changelog
-See [CHANGELOG.md](../CHANGELOG.md) for all changes per version.
+See [CHANGELOG.md](CHANGELOG.md) for all changes per version.
 
 ## Contributing
 See [CONTRIBUTING.md](../CONTRIBUTING.md) for more information on how to contribute to this project.