Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for minio and azurite for cloud sample #297

Merged
merged 7 commits into from
Aug 29, 2024

Conversation

u-veys
Copy link
Contributor

@u-veys u-veys commented Aug 2, 2024

What this PR changes/adds

It changes the previous transfers so that a transfer between two docker containers is possible. A file is transferred from azurite to minio.
To start the containers, a docker-compose is used, which starts the containers on the necessary ports with the necessary login data. Here, azurite acts as the provider and minio as the consumer.
Once the containers have been started successfully, a blob storage is created using the Amazon CLI and the test-document is added to it.
The hashicorp-vault is used to store the login data. A vault server must be started manually and the keys added.

The azure-storage-dataplane is used to load the file from azurite, while the aws-s3-dataplane is used to send it to the consumer.

These changes have been added in a new module transfer-06-file-transfer-docker. The jsons for the transfer as well as the docker-compose and the test-document can be found in the resources folder.

Why it does that

For making the transfer-05 runnable without need for cloud accounts.

Further notes

In addition to the new module, a change has been made to the gradle/libs.version.toml file. The hashicorp vault dependency has been added here.

Linked Issue(s)

Closes #248

@u-veys u-veys requested a review from ronjaquensel as a code owner August 2, 2024 14:00
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'We are always happy to welcome new contributors ❤️ To make things easier for everyone, please - make sure to follow our contribution guidelines, - check if you have already signed the ECA, and - relate this pull request to an existing issue or discussion.'

@ndr-brt ndr-brt self-requested a review August 2, 2024 14:33
Copy link
Member

@ndr-brt ndr-brt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please substitute this to the transfer-05, because at this point it become pointless.
An automated test in system-tests is mandatory to protect sample functionality against regressions

Copy link

This pull request is stale because it has been open for 7 days with no activity.

@github-actions github-actions bot added the stale Open for x days with no activity label Aug 13, 2024
@u-veys
Copy link
Contributor Author

u-veys commented Aug 13, 2024

I am currently working on the test, but I cannot start any tests. When starting the test I get the following error:

No matching tests found in any candidate test task.
    Requested tests:
        Test pattern org.eclipse.edc.samples.transfer.Transfer05fileTransferCloudTest in task :system-tests:test

@github-actions github-actions bot removed the stale Open for x days with no activity label Aug 14, 2024
@ndr-brt
Copy link
Member

ndr-brt commented Aug 14, 2024

I am currently working on the test, but I cannot start any tests. When starting the test I get the following error:

No matching tests found in any candidate test task.
    Requested tests:
        Test pattern org.eclipse.edc.samples.transfer.Transfer05fileTransferCloudTest in task :system-tests:test

if you added a tag annotation (like @EndToEndTest) you need to tell junit to use that tag.
we have an all-pass argument that's: -DrunAllTests=true (if you are using intellij idea you can add it on the gradle template so it will be added on every new test execution)

@ndr-brt ndr-brt added the enhancement New feature or request label Aug 19, 2024
@u-veys u-veys closed this Aug 26, 2024
@u-veys
Copy link
Contributor Author

u-veys commented Aug 26, 2024

Support for mino and azurite is now included directly in transfer-05. In addition, the system test is now also implemented. Maybe changes are still necessary, e.g. the minio bucket could be created in a different way. If changes are still necessary, I could take care of this

@u-veys u-veys reopened this Aug 26, 2024
@ndr-brt ndr-brt self-requested a review August 26, 2024 12:39
Copy link
Member

@ndr-brt ndr-brt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool stuff!

@u-veys
Copy link
Contributor Author

u-veys commented Aug 29, 2024

I have edited the requested changes. I also made another change in system-test 05 because the wrong vault-port was used in the test.

@ndr-brt ndr-brt merged commit 756484a into eclipse-edc:main Aug 29, 2024
6 checks passed
@3rim
Copy link

3rim commented Sep 5, 2024

Hello @u-veys , @ndr-brt

I try to understand why and how the HashiCorpVault is used here in this sample. Can someone explain?

kkakui pushed a commit to nttdata-dataspace/EDC-Samples-DAPS that referenced this pull request Sep 11, 2024
…#297)

* add minio and azurite support to cloud sample

* delete the vault connection on consumer side, since it is not needed

* system test for transfer-05 added

* minoi volume removed, bucket region changed and readme small change

* readme changes, unnecessary configs removed, system-test-05 wrong vault port fix and other small changes

* readme link to previous chapter fix

* readme vault installation added
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make transfer-05 runnable without the need for cloud accounts
3 participants