Does remote data passes through the provider connector?

[mujina93]

Hi,

general question, which I could not find the answer to by reading the documentation, nor by trying myself (because I am having problems in making a provider-consumer example work): when one requests remote data from a provider, will the provider return the url at which the resource can be found? (For example, the API url for open weather data from the example in the documentation), or will the data be queried by and funneled through the provider, which will send it over?

In other words, does data always passes through a provider (in case it's not local data)? And if so, what are the limits on scalability?

[juliapampus]

No, the provider will not return any access information. The data will be accessed and passed through the provider connector. If you want to handover the data transfer to another protocol, you would have to e.g. exchange the access information as the artifact's data to the data consumer. Then, this could somehow estabilish a connection to the provider's backend.

And if so, what are the limits on scalability? @brianjahnke

[mujina93]

Thank you for your reply.

In my question I meant "provider connector" when saying "provider".

So, do I understand correctly that if I return remote data via a provider connector, that data will be fetched by the provider connector and will be sent over to the provider?

Otherwise, I have to use my "own custom way" to send data over, which could be to store some access information (e.g. some external API url + credentials) as local data in the provider connector.

Did I understand correctly what you said?

If so, doesn't this defeats the purpose or exits a bit too much from the envisioned IDS architecture and connector usage?

[HeinrichPet]

@mujina93 Yes, the provider connector fetch the data from the remote URL and transfer it to the consumer. That was not done to be fast or scalable but to ensure that the provider and consumer connector have full control over the data and can enforce usage policies.

You can of course bypass this for example by switching to a separate channel via sharing credentials, but then you have no usage control trough the separate data exchange. So, yes this is a possibility but you leave then the IDS and its value proposition.

[juliapampus]

@mujina93 Yes, the provider connector fetch the data from the remote URL and transfer it to the consumer. That was not done to be fast or scalable but to ensure that the provider and consumer connector have full control over the data and can enforce usage policies.

You can of course bypass this for example by switching to a separate channel via sharing credentials, but then you have no usage control trough the separate data exchange. So, yes this is a possibility but you leave then the IDS and its value proposition.