Automatically populate PropertySource with instance metadata when run…

…ning within an EC2-based environment (#962)

---------

Co-authored-by: Maciej Walkowiak <walkowiak.maciej@yahoo.com>

docs/src/main/asciidoc/imds.adoc

@@ -0,0 +1,51 @@
+[#spring-cloud-aws-imds]
+== Instance Metadata Service Integration
+
+Spring Cloud AWS applications can use the Instance MetaData Service (IMDS) to acquire EC2 instance metadata when running within an EC2-based compute environment.  This metadata can be used for a wide variety of reasons, including detecting the availability zone, public IP address, MAC address, and so on.  When available, properties can be referenced using the @Value annotation:
+
+[source,java]
+----
+@Value("placement/availability-zone") String availabilityZone;
+@Value("public-ipv4") String publicIPAddress;
+@Value("mac") String macAddress;
+----
+
+A full list of instance metadata tags is available in the AWS reference documentation at link:https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-categories.html[AWS EC2 User Guide - Instance Metadata Categories]. Spring Cloud AWS always retrieves the "latest" categories of metadata and removes the prefix so that "/latest/meta-data/instance-id" is available as "instance-id".  The "spring.cloud.aws" prefix is also omitted. 
+
+=== Enabling
+
+To enable instance metadata, add the spring-cloud-aws-starter-imds starter.  
+
+[source,xml]
+----
+<dependency>
+    <groupId>io.awspring.cloud</groupId>
+    <artifactId>spring-cloud-aws-starter-imds</artifactId>
+</dependency>
+----
+
+This adds the software.amazon.awssdk/imds dependency to the classpath which is used to query the IMDS.  Depending on resources, metadata loading can add a half-second delay to application start time.  Loading can be explicitly disabled by setting spring.cloud.aws.imds.enabled propery:  
+
+[source,properties]
+----
+spring.cloud.aws.imds.enabled=false
+----
+
+Instance metadata is generally available on any EC2-based compute environment, which includes EC2, Elastic Beanstalk, Elastic Container Service (ECS), Elastic Kubernetes Service (EKS), etc.  It is not available in non-EC2 environments such as Lambda or Fargate.  Even within EC2-based compute environments instance metadata may be disabled or may be subject to an internal firewall which prohibits it.  Whenever instance metadata is unavailable, including when running on a local environment, the autoconfiguration process silently ignores its absence.
+
+=== Considerations
+
+Instance metadata is retrieved on a best effort basis and not all keys are always available.  For example, the "ipv6" key would only be present if IPv6 addresses were being used, "public-hostname" would only be available for instances running in public subnets with DNS hostnames enabled.
+
+Instance metadata is retrieved at application start time and is not updated as the application runs.  Both IDMS v1 and v2 are supported.  Certain keys / ranges are not retrieved, including "block-device-mapping/\*", "events/\*", "iam/security-credentials/\*", "network/interfaces/\*", "public-keys/\*", "spot/\*" for various reasons including security.  For example, Some keys such as "spot/termination-time" are only reliable if polled on an interval; presenting their static values obtained at startup time would be deceptive.  If you have such a requirement, consider polling the key yourself using the Ec2MetadataClient from the SDK: 
+
+[source,java]
+----
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.imds.Ec2MetadataClient;
+import software.amazon.awssdk.imds.Ec2MetadataResponse;
+...
+@Autowired Ec2MetadataClient client;
+client.get("/latest/meta-data/spot/termination-time");
+----
+

docs/src/main/asciidoc/index.adoc

@@ -101,6 +101,10 @@ A summary of these artifacts are provided below.
 | Provides integrations with SQS
 | io.awspring.cloud:spring-cloud-aws-starter-sqs
 
+| IMDS
+| Automatically loads EC2 instance metadata when running within an EC2-based environment
+| io.awspring.cloud:spring-cloud-aws-starter-imds
+
 | Parameter Store
 | Provides integrations with AWS Parameter Store
 | io.awspring.cloud:spring-cloud-aws-starter-parameter-store
@@ -144,6 +148,8 @@ include::sns.adoc[]
 
 include::sqs.adoc[]
 
+include::imds.adoc[]
+
 include::secrets-manager.adoc[]
 
 include::parameter-store.adoc[]

spring-cloud-aws-autoconfigure/pom.xml

@@ -34,6 +34,13 @@
 			<artifactId>spring-cloud-aws-core</artifactId>
 			<optional>true</optional>
 		</dependency>
+
+		<dependency>
+			<groupId>software.amazon.awssdk</groupId>
+			<artifactId>imds</artifactId>
+			<optional>true</optional>
+		</dependency>
+
 		<dependency>
 			<groupId>org.springframework.boot</groupId>
 			<artifactId>spring-boot-autoconfigure-processor</artifactId>
@@ -124,6 +131,11 @@
 			<artifactId>junit-jupiter</artifactId>
 			<scope>test</scope>
 		</dependency>
+		<dependency>
+			<groupId>org.wiremock</groupId>
+			<artifactId>wiremock-standalone</artifactId>
+			<scope>test</scope>
+		</dependency>
 		<dependency>
 			<groupId>io.micrometer</groupId>
 			<artifactId>micrometer-registry-cloudwatch2</artifactId>

spring-cloud-aws-autoconfigure/src/main/java/io/awspring/cloud/autoconfigure/imds/ImdsAutoConfiguration.java

@@ -0,0 +1,59 @@
+/*
+ * Copyright 2013-2022 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.awspring.cloud.autoconfigure.imds;
+
+import org.springframework.boot.autoconfigure.AutoConfiguration;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.context.annotation.Bean;
+import org.springframework.core.env.ConfigurableEnvironment;
+import software.amazon.awssdk.imds.Ec2MetadataClient;
+
+/**
+ * Configuration for managing Instance Meta Data Service metadata.
+ *
+ * @author Ken Krueger
+ * @since 3.1.0
+ */
+
+@AutoConfiguration
+@ConditionalOnClass(Ec2MetadataClient.class)
+@ConditionalOnProperty(name = "spring.cloud.aws.imds.enabled", havingValue = "true", matchIfMissing = true)
+public class ImdsAutoConfiguration {
+
+	@Bean
+	@ConditionalOnMissingBean
+	public ImdsPropertySource imdsPropertySource(ConfigurableEnvironment env, ImdsUtils imdsUtils) {
+		ImdsPropertySource propertySource = new ImdsPropertySource("Ec2InstanceMetadata", imdsUtils);
+		propertySource.init();
+		env.getPropertySources().addFirst(propertySource);
+		return propertySource;
+	}
+
+	@Bean
+	@ConditionalOnMissingBean
+	public ImdsUtils imdsUtils(Ec2MetadataClient ec2MetadataClient) {
+		return new ImdsUtils(ec2MetadataClient);
+	}
+
+	@Bean
+	@ConditionalOnMissingBean
+	public Ec2MetadataClient ec2MetadataClient() {
+		return Ec2MetadataClient.create();
+	}
+
+}

spring-cloud-aws-autoconfigure/src/main/java/io/awspring/cloud/autoconfigure/imds/ImdsPropertySource.java

@@ -0,0 +1,66 @@
+/*
+ * Copyright 2013-2022 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.awspring.cloud.autoconfigure.imds;
+
+import io.awspring.cloud.core.config.AwsPropertySource;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.springframework.lang.Nullable;
+
+/**
+ * Adds properties from the EC2 Instance MetaData Service (IDMS) when it is available.
+ *
+ * @author Ken Krueger
+ * @since 3.1.0
+ */
+public class ImdsPropertySource extends AwsPropertySource<ImdsPropertySource, ImdsUtils> {
+
+	private final String context;
+
+	private final ImdsUtils imdsUtils;
+
+	private final Map<String, String> properties = new LinkedHashMap<>();
+
+	public ImdsPropertySource(String context, ImdsUtils imdsUtils) {
+		super(context, imdsUtils);
+		this.context = context;
+		this.imdsUtils = imdsUtils;
+	}
+
+	@Override
+	public ImdsPropertySource copy() {
+		return new ImdsPropertySource(context, source);
+	}
+
+	@Override
+	public void init() {
+		if (!imdsUtils.isRunningOnCloudEnvironment())
+			return;
+		properties.putAll(imdsUtils.getEc2InstanceMetadata());
+	}
+
+	@Override
+	public String[] getPropertyNames() {
+		return properties.keySet().stream().toArray(String[]::new);
+	}
+
+	@Override
+	@Nullable
+	public Object getProperty(String name) {
+		return properties.get(name);
+	}
+
+}

spring-cloud-aws-autoconfigure/src/main/java/io/awspring/cloud/autoconfigure/imds/ImdsUtils.java

@@ -0,0 +1,134 @@
+/*
+ * Copyright 2013-2022 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package io.awspring.cloud.autoconfigure.imds;
+
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.core.exception.SdkClientException;
+import software.amazon.awssdk.imds.Ec2MetadataClient;
+import software.amazon.awssdk.imds.Ec2MetadataResponse;
+
+/**
+ * Utility object for working with EC2 instance metadata service (IDMS). Determines if the service and its data are
+ * available and provides utility methods for loading a PropertySource. Instance metadata is generally available - with
+ * exceptions - when an application is running within EC2, Elastic Beanstalk, ECS, EKS, etc. The presence of instance
+ * metadata can be used as a general indicator of whether an application is running within the AWS cloud or on a local
+ * environment, however there are exceptions to the general principal, such as when EC2 instance deliberately disables
+ * IMDS. Non-EC2 compute environments such as Lambda or Fargate do not provide the IMDS service.
+ *
+ * Works with either IMDS v1 or v2.
+ *
+ * @author Ken Krueger
+ * @since 3.1.0
+ */
+public class ImdsUtils {
+
+	private static Logger logger = LoggerFactory.getLogger(ImdsUtils.class);
+
+	private final Ec2MetadataClient client;
+
+	private Boolean isCloudEnvironment;
+
+	private final String prefix = "/latest/meta-data/";
+
+	private final String[] keys = { "ami-id", "ami-launch-index", "ami-manifest-path", "hostname", "instance-action",
+			"instance-id", "instance-life-cycle", "instance-type", "local-hostname", "local-ipv4", "mac", "profile",
+			"public-hostname", "public-ipv4", "reservation-id", "security-groups", "ipv6", "kernel-id", "iam/info",
+			"product-codes", "ramdisk-id", "reservation-id", "services/domain", "services/partition", "tags/instance",
+			"autoscaling/target-lifecycle-state", "placement/availability-zone", "placement/availability-zone-id",
+			"placement/group-name", "placement/host-id", "placement/partition-number", "placement/region"
+
+	};
+
+	public ImdsUtils(Ec2MetadataClient client) {
+		this.client = client;
+	}
+
+	public boolean isRunningOnCloudEnvironment() {
+		if (isCloudEnvironment == null) {
+			isCloudEnvironment = false;
+			try {
+				Ec2MetadataResponse response = client.get("/latest/meta-data/ami-id");
+				isCloudEnvironment = response.asString() != null && response.asString().length() > 0;
+			}
+			catch (SdkClientException e) {
+				if (e.getMessage().contains("retries")) {
+					// Ignore any exceptions about exceeding retries.
+					// This is expected when instance metadata is not available.
+				}
+				else {
+					logger.debug("Error occurred when accessing instance metadata.", e);
+				}
+			}
+			catch (Exception e) {
+				logger.error("Error occurred when accessing instance metadata.", e);
+			}
+			finally {
+				if (isCloudEnvironment) {
+					logger.info("EC2 Instance MetaData detected, application is running within an EC2 instance.");
+				}
+				else {
+					logger.info(
+							"EC2 Instance MetaData not detected, application is NOT running within an EC2 instance.");
+				}
+			}
+		}
+		return isCloudEnvironment;
+	}
+
+	/**
+	 * Load EC2 Instance Metadata into a simple Map structure, suitable for use populating a PropertySource.
+	 * @return Map of metadata properties. Empty Map if instance metadata is not available.
+	 *
+	 * @see ImdsPropertySource
+	 * @see ImdsAutoConfiguration
+	 */
+	public Map<String, String> getEc2InstanceMetadata() {
+		Map<String, String> properties = new LinkedHashMap<>();
+		if (!isRunningOnCloudEnvironment())
+			return properties;
+
+		Arrays.stream(keys).forEach(t -> mapPut(properties, t));
+
+		return properties;
+	}
+
+	/**
+	 * Internal utility method for safely loading a candidate key into the given Map. Silently ignores various expected
+	 * cases where keys are not present.
+	 * @param map
+	 * @param key
+	 */
+	private void mapPut(Map<String, String> map, String key) {
+		try {
+			Ec2MetadataResponse response = client.get(prefix + key);
+			if (response != null) {
+				map.put(key, response.asString());
+			}
+		}
+		catch (SdkClientException e) {
+			logger.debug("Unable to read property " + prefix + key + ", exception message: " + e.getMessage());
+		}
+		catch (RuntimeException e) {
+			logger.debug(
+					"Exception occurred reading property " + prefix + key + ", exception message: " + e.getMessage());
+		}
+	}
+
+}

spring-cloud-aws-autoconfigure/src/main/java/io/awspring/cloud/autoconfigure/imds/package-info.java

@@ -0,0 +1,22 @@
+/*
+ * Copyright 2013-2022 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Auto-configuration for IMDS (EC2 Instance MetaData Service) integration.
+ */
+@org.springframework.lang.NonNullApi
+@org.springframework.lang.NonNullFields
+package io.awspring.cloud.autoconfigure.imds;

spring-cloud-aws-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports

@@ -1,6 +1,7 @@
 io.awspring.cloud.autoconfigure.core.AwsAutoConfiguration
 io.awspring.cloud.autoconfigure.core.CredentialsProviderAutoConfiguration
 io.awspring.cloud.autoconfigure.core.RegionProviderAutoConfiguration
+io.awspring.cloud.autoconfigure.imds.ImdsAutoConfiguration
 io.awspring.cloud.autoconfigure.metrics.CloudWatchExportAutoConfiguration
 io.awspring.cloud.autoconfigure.ses.SesAutoConfiguration
 io.awspring.cloud.autoconfigure.s3.S3TransferManagerAutoConfiguration