Merge pull request #536 from Boavizta/532-document-how-cloud-scanner-provides-estimations-for-block-storage-ebs

535 and 532 Support AWS io1 io2 and standard EBS volumes, update doc.

Author: demeringo

CHANGELOG.md

@@ -9,8 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 _This paragraph may describe WIP/unreleased features. They are merged to main branch but not tagged._
 
-- https://github.com/Boavizta/cloud-scanner/issues/520 JSON inventory: move the execution statistics to the metadata section
-- https://github.com/Boavizta/cloud-scanner/issues/519 Add cloud-scanner version to inventory metadata
+- [JSON inventory: move the execution statistics to the metadata section. · Issue #520 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/520)
+- [Add cloud-scanner version to inventory metadata · Issue #519 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/519)
+- [Document how cloud scanner provides estimations for Block storage (EBS) · Issue #532 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/532)
+- [Support AWS io1 io2 and standard EBS volumes · Issue #535 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/535)
 
 ## [3.0.0]-2024-06-05
 

cloud-scanner-cli/src/aws_cloud_provider.rs

@@ -511,6 +511,6 @@ mod tests {
         let aws: AwsCloudProvider = AwsCloudProvider::new("eu-west-1").await.unwrap();
         let filtertags: Vec<String> = Vec::new();
         let res = aws.list_volumes(&filtertags).await.unwrap();
-        assert_eq!(4, res.len());
+        assert_eq!(6, res.len());
     }
 }

cloud-scanner-cli/src/boavizta_api_v1.rs

@@ -92,7 +92,7 @@ impl BoaviztaApiV1 {
                 };
 
                 match storage_type.as_str() {
-                    "st1" | "sc1" => {
+                    "st1" | "sc1" | "standard" => {
                         // This is a HDD
                         let res = component_api::disk_impact_bottom_up_v1_component_hdd_post(
                             &self.configuration,
@@ -107,14 +107,14 @@ impl BoaviztaApiV1 {
                             Ok(res) => Some(res),
                             Err(e) => {
                                 warn!(
-                                    "Warning: Cannot get HHD impact from API for type {}: {}",
+                                    "Warning: Cannot get HHD impact from API for storage type {}: {}",
                                     storage_type, e
                                 );
                                 None
                             }
                         }
                     }
-                    "gp2" | "gp3" => {
+                    "gp2" | "gp3" | "io1" | "io2" => {
                         // Use impacts of an SSD
                         let res = component_api::disk_impact_bottom_up_v1_component_ssd_post(
                             &self.configuration,
@@ -129,7 +129,7 @@ impl BoaviztaApiV1 {
                             Ok(res) => Some(res),
                             Err(e) => {
                                 warn!(
-                                    "Warning: Cannot get SSD impact from API for type {}: {}",
+                                    "Warning: Cannot get SSD impact from API for storage type {}: {}",
                                     storage_type, e
                                 );
                                 None
@@ -138,7 +138,7 @@ impl BoaviztaApiV1 {
                     }
                     _ => {
                         warn!(
-                            "Unknown disk type ({:?}), we use impacts of an ssd {:?}",
+                            "Unknown storage type ({:?}), defaulting to using impacts of an SSD {:?}",
                             storage_type.as_str(),
                             disk
                         );

docs/src/SUMMARY.md

@@ -11,6 +11,7 @@
 
 - [Methodology](explanations/methodology.md)
 - [How we process workload](explanations/processing-workload.md)
+- [Estimating Block Storage ](explanations/block-storage.md)
 
 # How-to guides
 
@@ -24,7 +25,7 @@
 - [Filtering by tags](how-to/filter-by-tags.md)
 - [Using a private instance of Boavizta API](how-to/using-private-boaviztapi.md)
 
-# Reference
+# Referen
 
 - [FAQ 💡](reference/FAQ.md)
 - [Common issues 🧰](reference/common-issues.md)

docs/src/explanations/block-storage.md

@@ -0,0 +1,36 @@
+# Estimating impacts of block storage
+
+Block storage volumes are approximated either as SSD or HDD depending on cloud provider specific storage type.
+
+This is a basic estimation. It takes into account the size of the block storage volume.
+
+⚠ Only the **manufacture** impacts of the Block storage are returned. The impacts of the **use** phase are returned as zero (mainly because we lack the related data in current version of Boavizta API).
+
+## Block storage (EBS) estimation (AWS)
+
+### What is considered in estimation:
+
+- storage type (HDD vs SSD)
+- storage size
+- duration of use
+
+| AWS EBS volume type | Boavizta storage used for estimation | comments |
+| :------------------ | :----------------------------------- | :------- |
+| st1                 | HDD                                  |          |
+| sc1                 | HDD                                  |          |
+| magnetic (standard) | HDD                                  |          |
+| gp2                 | SSD                                  |          |
+| gp3                 | SSD                                  |          |
+| io1                 | SSD                                  |          |
+| io2                 | SSD                                  |          |
+| unknown type        | SSD                                  | If it cannot identify the type of storage (like when a new type is introduced, cloud scanner uses SSD as an approximation)         |
+
+### What is not considered for the estimation
+
+- If the storage offers provisioned IOPS or not.
+- Implicit redundancy related to storage type.
+- Any other infrastructure (network, backup, supervision or redundancy) that may allow the cloud provider to offer EBS service.
+
+## Block storage estimations (Azure)
+
+Not yet implemented.

docs/src/explanations/methodology.md

@@ -1,21 +1,35 @@
 # Methodology and source of data
 
-Cloud scanner uses the Boavizta methodology to estimate the impacts of cloud resources.
+Cloud scanner relies on the Boavizta methodology and data to estimate the impacts of cloud resources.
 
-## Source of impact data
+It performs an inventory of the resources used in a cloud account. It then returns impacts of the *use* phase but also impacts related to the *manufacture* of the identified resources (*embodied* impacts).
+
+## The source of impact data
 
 Impact data is retrieved from [BOAVIZTA reference data API](https://github.com/Boavizta/boaviztapi/) v1.2.x.
 
+## General approach and limits
+
+Cloud scanner estimates impacts according to the duration of use of cloud resources. It also considers the "intensity" of use (like the type of instance and CPU load for a VM or size of a disk for storage). The region of use is also considered to take into account the electricity mix of the region (carbon intensity).
+
+The general approach is to map cloud vendor specific ressources, either to Boavizta *cloud resources* (like for VM instances) or to more generic components (Block storage volumes are approximated as Boavizta HDD or SSD).
+
+But this approach means that **a large portion** of the cloud provider infrastructure **is excluded from the estimation**.
+
+⚠ Cloud scanner **underestimates the impacts of a cloud account**. Because it only considers the _instances_ and _block storage_, many additional sources of impacts (network, potential redundancy, cloud control plan, buildings...) are not included in the estimation.
+
+![A diagram that show that only VM's and Block storage are considered](../images/cloud-scanner-perimeter-aws.excalidraw.png)
+
+See also [other limits](../reference/limits.md).
+
 ## Methodology
 
-The general approach of Boavizta is described in [Digital & environment : How to evaluate server manufacturing footprint, beyond greenhouse gas emissions? | Boavizta](https://boavizta.org/en/blog/empreinte-de-la-fabrication-d-un-serveur)
+The Boavizta methodology is described in [Digital & environment : How to evaluate server manufacturing footprint, beyond greenhouse gas emissions? | Boavizta](https://boavizta.org/en/blog/empreinte-de-la-fabrication-d-un-serveur)
 
 The impacts (use and embedded) are attributed according to the principles described in [Cloud instances - Boavizta API documentation](https://doc.api.boavizta.org/Explanations/services/cloud/).
 
-The results are similar to what you can visualize in [Datavizta](http://datavizta.boavizta.org/cloudimpact), but with automated inventory.
+The results of cloud scanner are similar to what you can visualize in [Datavizta](http://datavizta.boavizta.org/cloudimpact), but with automated inventory.
 
-⚠ Cloud scanner **underestimates the impacts of the cloud resources**. Because it only considers the _instances_ and _block storage_ a several sources of impacts (network, potential redundancy, cloud control plan) are not included in the estimation.
 
-See also [other limits](../reference/limits.md).
+- Boavizta website: <https://www.boavizta.org/en>
 
-- https://www.boavizta.org/en

docs/src/explanations/processing-workload.md

@@ -1,12 +1,20 @@
-# How we process workload
+# How we estimate instance workload
 
-Workload of instances are estimated using AWS cloudwatch CPU metrics summary.
+Workload (or intensity of use) of instances is estimated using CPU load level as a proxy.
+
+## Estimating instances workload for AWS
+
+The CPU load of AWS instances is retrieved using _AWS Cloudwatch CPU metrics summary_.
 
 Cloud scanner uses a sampling period of 15 minutes, but impacts metrics are returned as impacts equivalent to one hour of use.
 
 This means that instance impacts metrics data returned can be understood as: `impact for one hour of use (considering  the CPU workload of 15 last minutes)`.
 
 Why this default sampling period of 15 minutes ?
 
-- It seems sufficient for our current monitoring needs (but maybe we can make it configurable in the future).
-- It seems hard to go below 10 minutes (because default period of AWS instance metrics is 5 minutes. You need to activate `detailed monitoring` (extra feature) for 1 minute granularity: [List the available CloudWatch metrics for your instances - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html#ec2-cloudwatch-metrics)).
+- It is sufficient for our current monitoring needs (but maybe we may make this setting configurable in the future).
+- It is hard to go below 10 minutes. Default (and free) period of AWS instance metrics is 5 minutes. You need to activate `detailed monitoring` (extra feature) for 1 minute granularity: [List the available CloudWatch metrics for your instances - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html#ec2-cloudwatch-metrics)).
+
+## Estimating instances workload for Azure
+
+Not implemented.