Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Client side S3EncryptionClient #1033

Merged
merged 25 commits into from
Nov 6, 2024
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

84 changes: 84 additions & 0 deletions docs/src/main/asciidoc/s3.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,90 @@ try (OutputStream outputStream = s3Resource.getOutputStream()) {
}
----

=== S3 Client Side Encryption
MatejNedic marked this conversation as resolved.
Show resolved Hide resolved

AWS offers encryption library which is integrated inside of S3 Client called https://docs.aws.amazon.com/amazon-s3-encryption-client/latest/developerguide/what-is-s3-encryption-client.html [S3EncryptionClient].
With encryption client you are going to encrypt your files before sending them to S3 bucket.

To autoconfigure Encryption Client simply add the following dependency.

[source,xml]
----
<dependency>
<groupId>software.amazon.encryption.s3</groupId>
<artifactId>amazon-s3-encryption-client-java</artifactId>
</dependency>
----


We are supporting 3 types of encryption.

1. To configure encryption via KMS key specify 'spring.cloud.aws.s3.encryption.keyId' with KMS key arn and this key will be used to encrypt your files.

Also, following dependency is required.
[source,xml]
----
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>kms</artifactId>
<optional>true</optional>
</dependency>
----


2. Asymmetric encryption is possible via RSA to enable it you will have to implement 'io.awspring.cloud.autoconfigure.s3.S3RsaProvider'

!Note you will have to manage storing private and public keys yourself otherwise you won't be able to decrypt the data later.
Example of simple RSAProvider:

[source,java,indent=0]
----
import io.awspring.cloud.autoconfigure.s3.S3RsaProvider;
import java.security.KeyPair;
import java.security.KeyPairGenerator;

public class MyRsaProvider implements S3RsaProvider {
@Override
public KeyPair generateKeyPair() {
MatejNedic marked this conversation as resolved.
Show resolved Hide resolved
try {
KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("RSA");
keyPairGenerator.initialize(2048);
return keyPairGenerator.generateKeyPair();
}
catch (Exception e) {
return null;
}
}
}
----

3. Last option is if you want to use symmetric algorithm, this is possible via `io.awspring.cloud.autoconfigure.s3.S3AesProvider`

!Note you will have to manage storing storing private key!
Example of simple AESProvider:

[source,java,indent=0]
----
import io.awspring.cloud.autoconfigure.s3.S3AesProvider;
import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;

public class MyAesProvider implements S3AesProvider {
@Override
public SecretKey generateSecretKey() {
try {
KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");
keyGenerator.init(256);
return keyGenerator.generateKey();
}
catch (Exception e) {
return null;
}
}
}
----


==== S3 Output Stream

Under the hood by default `S3Resource` uses a `io.awspring.cloud.s3.InMemoryBufferingS3OutputStream`. When data is written to the resource, is gets sent to S3 using multipart upload.
Expand Down
10 changes: 10 additions & 0 deletions spring-cloud-aws-autoconfigure/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -166,5 +166,15 @@
<artifactId>sts</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>kms</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>software.amazon.encryption.s3</groupId>
<artifactId>amazon-s3-encryption-client-java</artifactId>
<optional>true</optional>
</dependency>
</dependencies>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/*
* Copyright 2013-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.awspring.cloud.autoconfigure.s3;

import javax.crypto.SecretKey;

public interface S3AesProvider {
MatejNedic marked this conversation as resolved.
Show resolved Hide resolved

SecretKey generateSecretKey();
}
Original file line number Diff line number Diff line change
Expand Up @@ -35,26 +35,27 @@
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.autoconfigure.condition.*;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.boot.context.properties.PropertyMapper;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.util.ClassUtils;
import org.springframework.util.StringUtils;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.regions.providers.AwsRegionProvider;
import software.amazon.awssdk.s3accessgrants.plugin.S3AccessGrantsPlugin;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3ClientBuilder;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.encryption.s3.S3EncryptionClient;

/**
* {@link EnableAutoConfiguration} for {@link S3Client} and {@link S3ProtocolResolver}.
*
* @author Maciej Walkowiak
* @author Matej Nedic
*/
@AutoConfiguration
@ConditionalOnClass({ S3Client.class, S3OutputStreamProvider.class })
Expand Down Expand Up @@ -85,6 +86,7 @@ S3ClientBuilder s3ClientBuilder(AwsClientBuilderConfigurer awsClientBuilderConfi
.enableFallback(properties.getPlugin().getEnableFallback()).build();
builder.addPlugin(s3AccessGrantsPlugin);
}

Optional.ofNullable(this.properties.getCrossRegionEnabled()).ifPresent(builder::crossRegionAccessEnabled);

builder.serviceConfiguration(this.properties.toS3Configuration());
Expand Down Expand Up @@ -121,10 +123,69 @@ else if (awsProperties.getEndpoint() != null) {

@Bean
@ConditionalOnMissingBean
@ConditionalOnMissingClass(value = { "software.amazon.encryption.s3.S3EncryptionClient" })
S3Client s3Client(S3ClientBuilder s3ClientBuilder) {
return s3ClientBuilder.build();
}

@Configuration
@ConditionalOnClass(name = { "software.amazon.encryption.s3.S3EncryptionClient" })
static class S3EncryptionConfiguration {

@Bean
@ConditionalOnMissingBean
S3Client s3EncryptionClient(S3EncryptionClient.Builder s3EncryptionBuilder, S3ClientBuilder s3ClientBuilder) {
s3EncryptionBuilder.wrappedClient(s3ClientBuilder.build());
return s3EncryptionBuilder.build();
}

@Bean
@ConditionalOnMissingBean
S3EncryptionClient.Builder s3EncrpytionClientBuilder(S3Properties properties,
AwsClientBuilderConfigurer awsClientBuilderConfigurer,
ObjectProvider<AwsClientCustomizer<S3EncryptionClient.Builder>> configurer,
ObjectProvider<AwsConnectionDetails> connectionDetails,
ObjectProvider<S3EncryptionClientCustomizer> s3ClientCustomizers,
ObjectProvider<AwsSyncClientCustomizer> awsSyncClientCustomizers,
ObjectProvider<S3RsaProvider> rsaProvider, ObjectProvider<S3AesProvider> aesProvider) {
S3EncryptionClient.Builder builder = awsClientBuilderConfigurer.configureSyncClient(
S3EncryptionClient.builder(), properties, connectionDetails.getIfAvailable(),
configurer.getIfAvailable(), s3ClientCustomizers.orderedStream(),
awsSyncClientCustomizers.orderedStream());

Optional.ofNullable(properties.getCrossRegionEnabled()).ifPresent(builder::crossRegionAccessEnabled);
builder.serviceConfiguration(properties.toS3Configuration());

configureEncryptionProperties(properties, rsaProvider, aesProvider, builder);
return builder;
}

private static void configureEncryptionProperties(S3Properties properties,
ObjectProvider<S3RsaProvider> rsaProvider, ObjectProvider<S3AesProvider> aesProvider,
S3EncryptionClient.Builder builder) {
PropertyMapper propertyMapper = PropertyMapper.get();
var encryptionProperties = properties.getEncryption();

propertyMapper.from(encryptionProperties::isEnableDelayedAuthenticationMode)
.to(builder::enableDelayedAuthenticationMode);
propertyMapper.from(encryptionProperties::isEnableLegacyUnauthenticatedModes)
.to(builder::enableLegacyUnauthenticatedModes);
propertyMapper.from(encryptionProperties::isEnableMultipartPutObject).to(builder::enableMultipartPutObject);

if (!StringUtils.hasText(properties.getEncryption().getKeyId())) {
if (aesProvider.getIfAvailable() != null) {
builder.aesKey(aesProvider.getObject().generateSecretKey());
}
else {
builder.rsaKeyPair(rsaProvider.getObject().generateKeyPair());
MatejNedic marked this conversation as resolved.
Show resolved Hide resolved
}
}
else {
propertyMapper.from(encryptionProperties::getKeyId).to(builder::kmsKeyId);
}
}
}

@Configuration
@ConditionalOnClass(ObjectMapper.class)
static class Jackson2JsonS3ObjectConverterConfiguration {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
/*
* Copyright 2013-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.awspring.cloud.autoconfigure.s3;

import io.awspring.cloud.autoconfigure.AwsClientCustomizer;
import software.amazon.encryption.s3.S3EncryptionClient;

/**
* Callback interface that can be used to customize a {@link S3EncryptionClient.Builder}.
*
* @author Matej Nedic
* @since 3.3.0
*/
@FunctionalInterface
public interface S3EncryptionClientCustomizer extends AwsClientCustomizer<S3EncryptionClient.Builder> {
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/*
* Copyright 2013-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.awspring.cloud.autoconfigure.s3;

import java.security.KeyPair;

public interface S3RsaProvider {

KeyPair generateKeyPair();
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/*
* Copyright 2013-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.awspring.cloud.autoconfigure.s3.properties;

/**
* Properties to configure {@link software.amazon.encryption.s3.S3EncryptionClient}
* @author Matej Nedic
*/
public class S3EncryptionProperties {
private boolean enableLegacyUnauthenticatedModes = false;
private boolean enableDelayedAuthenticationMode = false;
private boolean enableMultipartPutObject = false;
private String keyId;

public String getKeyId() {
return keyId;
}

public void setKeyId(String keyId) {
this.keyId = keyId;
}

public boolean isEnableLegacyUnauthenticatedModes() {
return enableLegacyUnauthenticatedModes;
}

public void setEnableLegacyUnauthenticatedModes(boolean enableLegacyUnauthenticatedModes) {
this.enableLegacyUnauthenticatedModes = enableLegacyUnauthenticatedModes;
}

public boolean isEnableDelayedAuthenticationMode() {
return enableDelayedAuthenticationMode;
}

public void setEnableDelayedAuthenticationMode(boolean enableDelayedAuthenticationMode) {
this.enableDelayedAuthenticationMode = enableDelayedAuthenticationMode;
}

public boolean isEnableMultipartPutObject() {
return enableMultipartPutObject;
}

public void setEnableMultipartPutObject(boolean enableMultipartPutObject) {
this.enableMultipartPutObject = enableMultipartPutObject;
}

}
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
/*
* Copyright 2013-2024 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.awspring.cloud.autoconfigure.s3.properties;

public class S3PluginProperties {
Expand Down
Loading