Data Transfer to FastAPI Data Sink Demo Setup (Production environment)

[DanieleCrippaIMECH]

Issue: UI data transfer setup with FastAPI data sink

Hello,

I'm currently setting up a production environment demo using EDC-CE and Sovity Authority Portal, in the context of SM4RTENANCE project.

The reverse proxy configuration for the required ports appears to be working correctly, as the Authority Portal shows a green light and I'm able to successfully negotiate contracts.

To test data transfer, I’ve implemented a simple FastAPI-based data sink. Both the FastAPI service and the connector are running on a VM within our company network.

The FastAPI application works as expected when tested from other machines on the local network using Postman.

However, when I attempt a data transfer via the UI, the process appears to complete successfully, but the FastAPI server is never triggered. Additionally, neither the connector nor the UI container logs show any activity related to the transfer.

As the endpoint for the data transfer, I'm simply providing the local endpoint in my POST request setup, for example:

http://192.168.1.100:5000/upload

assuming that, since both the connector and the FastAPI data sink are running on the same machine, this would be a valid reference. However, I’m starting to wonder if that assumption is incorrect. Could you help me understand whether this setup is valid, or if I should be using a different address or approach?

Also, for the connector-data sink communication, no specific header or authentication is needed at the moment.

Components

Sovity EDC-CE v10.4.2
Sovity EDC-UI v4.1.6
Sovity Authority Portal

FastAPI request

from fastapi import FastAPI, UploadFile, File, HTTPException, Request
from fastapi.responses import JSONResponse, FileResponse
import os
import json
import uvicorn
from fastapi import status

UPLOAD_FOLDER = "uploads"
os.makedirs(UPLOAD_FOLDER, exist_ok=True)

app = FastAPI()

@app.post("/upload")
async def upload_json(request: Request, filename: str = "uploaded.json"):
    try:
        content_type = request.headers.get("content-type")
        
        if content_type == "application/json":
            data = await request.json()
        elif content_type == "application/octet-stream":
            raw_data = await request.body()
            try:
                data = json.loads(raw_data.decode("utf-8"))
            except json.JSONDecodeError:
                raise HTTPException(status_code=400, detail="Invalid JSON in octet-stream")
        else:
            raise HTTPException(status_code=415, detail=f"Unsupported Content-Type: {content_type}")

        filepath = os.path.join(UPLOAD_FOLDER, filename)
        with open(filepath, "w") as f:
            json.dump(data, f)

        return JSONResponse(content={"message": "File uploaded successfully", "filename": filename}, status_code=200)
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

if __name__ == "__main__":
    uvicorn.run("app:app", host="0.0.0.0", port=5000, reload=True)

[tmberthold]

Hello and thank you for opening this discussion 🚀

To identify whether it is due to the sink or source, you could use a demo sink known to work, e.g. a webhooksite as linked below, and see whether a GET call for the request to the data source arrives when initiating the transfer and later a POST call for posting the data to the sink. Both can be tried via the webhook site, then you just have 2 calls there in the history, once for the quasi source and once for the quasi sink.

Link: https://webhook.site/

[DanieleCrippaIMECH]

Hello and thank you for the suggestion.

When I try to use the webhook as a data sink, there is no call in the history, so it seems it is not being reached, similarly to my FastAPI data sink.

[tmberthold]

Thank you for the update. A few follow-up points:

• Webhook as Data Source: Have you had a chance to configure a webhook as a Data Source, as suggested earlier? It would be great to confirm whether this approach is now in place to confirm if it is working with a non-local setup.
• Logging: Whenever the EDC cannot reach the Data-Source or -Sink during an initiated transfer, there are logs. In particular, the UI itself will at least emit logs upon access at the latest at the proxy-level.
• Production Accessibility: For any production-grade deployment as here for the production environment demo, all components must be reachable from the Internet. Running parts of the system on a local machine will not suffice - mixed local/in-cloud configurations are unlikely to function correctly under real-world conditions.
• Production Setup Guide: For detailed instructions on deploying the Community Edition in a production environment, please refer to our guide: https://edc-ce.docs.sovity.de/community-edition-deployment/production-ce

If this still doesn't help, I will be happy to pass this on to our EDC experts. Preliminary ping already now for context awareness: @richardtreier @ununhexium

[DanieleCrippaIMECH]

Hello,

I’ve configured a webhook as a data source, and it appears to be working correctly (green light on the portal data catalogue). However, using webhook as a data sink seems problematic: although the interface reports a successful data transfer, there are no actual incoming requests to the webhook, similar to what happens with my local data sink.

I’ve configured my local connector accordingly to the documentation. To avoid potential local connectivity issues, I exposed the sink via a public HTTPS endpoint and used this in the request settings through the UI. Unfortunately, the behavior remains the same: the transfer is reported as successful, but no data is received.

Additionally, I see no relevant logs when initiating the transfer — neither from the UI nor from the connector containers (using "docker logs <container_name>"). The nginx reverse proxy logs show no errors, but there is some indication of effective transfer initialization, as confirmed by the following logs:

"POST /api/dsp/catalog/request HTTP/1.1" 200 1446 "-" "okhttp/4.12.0"
"POST /api/dsp/transfers/aec3d2ce-96ea-4a4e-b2df-4b7a48161b2a/start HTTP/1.1" 200 5 "-" "okhttp/4.12.0"

Would you have any suggestions on further diagnostics or configuration checks?

[tmberthold]

Thank you for the details.

Just to confirm: when initiating the transfer with the webhook site as a data-source, is a GET request also visible/listed at the webhook site, which would confirm the transfer call reaching the data-source?

If not, it would indicate that in your setup the data-source is not being properly addressed, which would also explain why the data-sink doesn't receive data, the green light alone is not sufficient to verify this.

Nevertheless, I will forward this internally as well.

[DanieleCrippaIMECH]

Thank you for your support.

Right now, I only have one connector and a demo data source for the SM4RTENANCE project, so I don’t have a consumer available to test if the connector works properly as a producer. For now, I can only confirm that it was accepted by the Authority Portal Data Catalogue when I configured a data asset using the webhook as the data source. We're working on setting up a second connector, and I’ll check this as soon as it’s ready.

[DanieleCrippaIMECH]

Using a second connector with a custom data source resolved the issue. It appears the problem was due to an empty data source: while the data transfer initiated correctly, the sink wasn't triggered since there was no data to transmit. Thanks again for your support.