diff --git a/README.md b/README.md index 6a0ada22..eeb89c7e 100644 --- a/README.md +++ b/README.md @@ -10,14 +10,14 @@ ### Summary -The **`gcamfaostat`** R package is designed for the preparation, processing, and synthesis of the Food and Agriculture Organization (FAO) Statistics ([FAOSTAT](https://www.fao.org/faostat/en/#data)) agroeconomic data. The primary purpose is to facilitate FAOSTAT data use in global economic and multisector dynamic models while ensuring transparency, traceability, and reproducibility. Here, we provide an overview of the development of **`gcamfaostat`** (v1.0) and demonstrate its capabilities in generating and maintaining agroeconomic data required for the Global Change Analysis Model ([GCAM](https://jgcri.github.io/gcam-doc/toc.html)). Our initiative seeks to enhance the quality and accessibility of data for the global agroeconomic modeling community, with the aim of fostering more robust and harmonized outcomes in a collaborative, efficient, and open-source framework. One of the important features of the package is the possibility to construct the FAO Food Balance Sheets at the disaggregated commodity level (with over 500 commodities), which provides a comprehensive and detailed data input for a variety of analytical and modeling applications. The processed data and visualizations offered by **`gcamfaostat`** can also be valuable to a broader audience interested in gaining insights into the intricacies of global agriculture. +The **`gcamfaostat`** R package is designed for the preparation, processing, and synthesis of the Food and Agriculture Organization (FAO) Statistics ([FAOSTAT](https://www.fao.org/faostat/en/#data)) agroeconomic data. The primary purpose is to facilitate FAOSTAT data use in global economic and multisector dynamic models while ensuring transparency, traceability, and reproducibility. Here, we provide an overview of the development of **`gcamfaostat`** (v1.0.0) and demonstrate its capabilities in generating and maintaining agroeconomic data required for the Global Change Analysis Model ([GCAM](https://jgcri.github.io/gcam-doc/toc.html)). Our initiative seeks to enhance the quality and accessibility of data for the global agroeconomic modeling community, with the aim of fostering more robust and harmonized outcomes in a collaborative, efficient, and open-source framework. One of the important features of the package is the possibility to construct the FAO Food Balance Sheets at the disaggregated commodity level (with over 500 commodities), which provides a comprehensive and detailed data input for a variety of analytical and modeling applications. The processed data and visualizations offered by **`gcamfaostat`** can also be valuable to a broader audience interested in gaining insights into the intricacies of global agriculture. This tool bridges a crucial gap in the literature by offering several key features and capabilities. 1. **Transparency and Reproducibility**: **`gcamfaostat`** incorporates functions for downloading, cleaning, synthesizing, and balancing agroeconomic datasets in a traceable, transparent, and reproducible manner. This enhances the credibility of the processing and allows for better scrutiny of the methods. We have documented and demonstrated the use of the package in generating and updating agroeconomic data needed for the GCAM. 2. **Expandability and Consistency**: **`gcamfaostat`** can be used to flexibly process and update agroeconomic data for any agroeconomic model. The package framework can be also easily expanded to include new modules for consistently processing new data. -3. **Community Collaboration and Efficiency**: The package provides an open-source platform for researchers to continually enhance the processing methods. This collaborative approach, which establishes a standardized and streamlined process for data preparation and processing, carries benefits that extend to all modeling groups. By reducing the efforts required for data processing and fostering harmonized base data calibration, it contributes to a reduction in modeling uncertainty and enhances the overall research efficiency. +3. **Community Collaboration and Efficiency**: The package provides an open-source platform for researchers to continually enhance the processing methods. This collaborative approach, which establishes a standardized and streamlined process for data preparation and processing, carries benefits that extend to all modeling groups. By reducing the effort required for data processing and fostering harmonized base data calibration, it contributes to a reduction in modeling uncertainty and enhances the overall research efficiency. 4. **User Accessibility**: Where applicable, the processed data can be mapped and aggregated to user-specified regions and sectors for agroeconomic modeling. However, beyond the modeling community, **`gcamfaostat`** can be valuable to a broader range of users interested in understanding global agriculture trends and dynamics, as it provides user-friendly data processing and visualization tools. *** @@ -82,7 +82,7 @@ Data development is never a once and for all task, and continued efforts are nee 1. **Sustain processing functions for updated raw data**: ensuring that our processing functions remain up-to-date when raw data undergoes revisions is imperative. 2. **Evaluate and enhance assumptions**: a critical examination of the assumptions utilized in processes like interpolation, extrapolation, aggregation, disaggregation, and mapping is essential and should be an ongoing endeavor. 3. **Revise assumptions in low-quality data zones**: regions and sectors with little or low-quality data require careful consideration. We will need to adjust our assumptions when improved data becomes available. -4. **Promoting broader applications**: leveraging data processed by gcamfaostat can significantly contribute to harmonizing input data in global agroeconomic modeling. Encouraging the utilization of this data and fostering collaboration to enhance data processing is of significant importance. +4. **Promoting broader applications**: leveraging data processed by gcamfaostat can significantly contribute to harmonizing input data in global agroeconomic modeling. Encouraging the utilization of this data and fostering collaboration to enhance data processing is crucial. 5. **Assess sensitivity in downstream applications**: understanding the sensitivity of downstream data applications, e.g., global agroeconomic projections, to upstream data processing assumptions is crucial. This awareness empowers us to make informed decisions and refinements. We welcome and value community contributions to gcamfaostat. Please read our [Contributing Guidelines](CONTRIBUTING.md) for information on how to contribute to this package. Through collective and collaborative efforts, we hope to improve the interface between raw data, modeling community, and broader audience. We would be grateful for the feedback and suggestions on potential improvements of the developed data processing framework. @@ -102,4 +102,4 @@ We welcome and value community contributions to gcamfaostat. Please read our [Co -Copyright 2019 Battelle Memorial Institute; see the LICENSE file. +Copyright 2023 Battelle Memorial Institute; see the LICENSE file. diff --git a/docs/articles/vignette_getting_started.html b/docs/articles/vignette_getting_started.html index 5ffc7481..d8339072 100644 --- a/docs/articles/vignette_getting_started.html +++ b/docs/articles/vignette_getting_started.html @@ -112,7 +112,7 @@

Getting Started with gcamfaostat

-

2023-11-07

+

2023-11-10

Source: vignettes/vignette_getting_started.Rmd @@ -159,7 +159,7 @@

Introduction (Bond-Lamberty et al. 2019). While largely overlooked, the uncertainty in the base data calibration approach likely -contributed to the disparities in model outcomes (Lampe et al. 2014; Zhao, Calvin, Wise, and Iyer +contribute to the disparities in model outcomes (Lampe et al. 2014; Zhao, Calvin, Wise, and Iyer 2021). Hence, our motivation is to create an open-source tool (gcamfaostat) for the preparation, processing, and synthesis of FAOSTAT data for global agroeconomic @@ -178,9 +178,9 @@

The gaps gcamfaostat(Bond-Lamberty et al. 2019). As an -illustration, in the latest GCAM version, GCAM v7 (Bond-Lamberty et al. 2023), about 280 XML +functions to convert raw data inputs into hundreds of XML input files +used by GCAM (Bond-Lamberty et al. 2019). +As an illustration, in the latest GCAM version, GCAM v7 (Bond-Lamberty et al. 2023), about 280 XML files, with a combined size of 4.1 GB, are generated. Although AgLU-related XMLs represent only about 10% of the total number of files, they contribute over 50% in size (~2.1 GB). The majority of AgLU-related @@ -195,10 +195,10 @@

The gaps gcamfaostat(Zhao and Wise -2023). The gcamfaostat package is -aimed to address these limitations (Figure +2023). The gcamfaostat package aims +to address these limitations (Figure 2). The targeted approach incorporates data preparation, -processing, and synthesizing capabilities within a dedicated package, +processing, and synthesis capabilities within a dedicated package, gcamfaostat, while regional and sectoral aggregation functions in the model data system are implemented using standalone routines within the gcamdata package. This strategy not only ensures the streamlined diff --git a/docs/articles/vignette_news.html b/docs/articles/vignette_news.html index 8ffd4f57..c3437571 100644 --- a/docs/articles/vignette_news.html +++ b/docs/articles/vignette_news.html @@ -112,7 +112,7 @@

News

-

2023-11-08

+

2023-11-10

Source: vignettes/vignette_news.Rmd @@ -122,15 +122,24 @@

2023-11-08

-

gcamfaostat 1.0.0 +

November 2023 updates

-

Beta release * The beta release of gcamfaostat 1.0.0 -includes the data generated for the Global Change Analysis Model v7.0 GCAM -v7.0. *

-

November 2023 updates +

+JOSS submission

+

2023-11-10

+
    +
  • gcamfaostat is now submitted to JOSS for review and publication
    • +
  • gcamfaostat v1.0.0 includes the data generated for the Global Change +Analysis Model v7.0 GCAM +v7.0.
    • +
  • We will make an official release after the JOSS review.
    • +
+

2023-11-09 # FAOSTAT (in the recent Oct release) provide 2021 data # +We will now update gcamfaostat to include the latest data and for GCAM +BYU # A new BYU branch is created for this task

2023-11-08

  • All Github Action test passed. Badges added!
    • @@ -199,7 +208,12 @@

    Goals for the next version include:<
  • Compare the results from the new primary commodity equivalent method with the FBS data compiled using FAOSTAT method (never disclosed).

-

Raw data archive 2023-08-19

+
+
+

+Raw data archive +

+

2023-08-19

The source data downloaded from FAOSTAT is archived at a Zenodo repository.

diff --git a/docs/articles/vignette_preparing_data.html b/docs/articles/vignette_preparing_data.html index f8235fe2..8e9340c1 100644 --- a/docs/articles/vignette_preparing_data.html +++ b/docs/articles/vignette_preparing_data.html @@ -113,7 +113,7 @@

Preparing data for gcamfaostat

-

2023-11-07

+

2023-11-10

Source: vignettes/vignette_preparing_data.Rmd diff --git a/docs/articles/vignette_processing_flow.html b/docs/articles/vignette_processing_flow.html index eb491c7e..3d0f131b 100644 --- a/docs/articles/vignette_processing_flow.html +++ b/docs/articles/vignette_processing_flow.html @@ -112,7 +112,7 @@

gcamfaostat data processing

-

2023-11-07

+

2023-11-10

Source: vignettes/vignette_processing_flow.Rmd @@ -152,19 +152,18 @@

Data synthesizing in a key moduleFigure 2. This endeavor entails working through nine tiers of data, each sourced differently, with the -aim of producing and harmonizing agricultural SUA dataset for a diverse -range of over 500 agricultural commodities. Compared to the FAO’s FBS, -which report food and nutritional information for about 100 composite -categories, in many cases combining primary and processed commodities -(e.g., wheat and flour) within a single category (e.g., wheat and -products), the constructed SUA explicitly trace the transformations -between primary and processed commodities, while reporting nutritional -details at the highly disaggregated level (over 500 commodities). In -doing so, the constructed dataset substantially simplifies the FAOSTAT -data processing steps by explicitly distinguishing food and nutritional -supply at the individual commodity level and facilitating a -straightforward mapping of the corresponding data to the global -agroeconomic models.

+aim of producing an harmonized agricultural SUA dataset for over 500 +agricultural commodities. Compared to the FAO’s FBS, which report food +and nutritional information for about 100 composite categories, in many +cases combining primary and processed commodities (e.g., wheat and +flour) within a single category (e.g., wheat and products), the +constructed SUA explicitly trace the transformations between primary and +processed commodities, while reporting nutritional details at a highly +disaggregated level (over 500 commodities). In doing so, the constructed +dataset substantially simplifies the FAOSTAT data processing steps by +explicitly distinguishing food and nutritional supply at the individual +commodity level and facilitating a straightforward mapping of the +corresponding data to the global agroeconomic models.

As an illustrative example, the first tier comprises 168 commodities, generated by combining production data from QCL, trade data from TM, and other essential balancing elements (such as opening and closing stocks, @@ -229,7 +228,8 @@

Examples 
 # Load gcamfaostat
-devtools::load_all() 
+# devtools::load_all() 
+library(gcamfaostat)
 # package data, GCAM_DATA_MAP, will be loaded
 
 # Get information for an intermediate data frame QCL_CROP_PRIMARY
diff --git a/docs/articles/vignette_use_cases.html b/docs/articles/vignette_use_cases.html
index 53f60cfa..a8dc9e04 100644
--- a/docs/articles/vignette_use_cases.html
+++ b/docs/articles/vignette_use_cases.html
@@ -112,7 +112,7 @@
     

       
Use Cases

             
-            
2023-11-07

+            
2023-11-10

       
       Source: vignettes/vignette_use_cases.Rmd
       
@@ -198,7 +198,7 @@ 
Country aggregation and disaggre
 functions to deal with changes in the country classifications.

 
FAOSTAT_AREA_RM_NONEXIST

 
    
-
  • The function removes nonexistent FAO region using area_code, e.g.,
+
  • The function removes nonexistent FAO regions using area_code, e.g.,
 USSR after 1991.
    
 
    • 
 
  • All nonexistent countries due to dissolution are removed by
diff --git a/docs/articles/vignette_visualization.html b/docs/articles/vignette_visualization.html
index c2f85c59..937b7d12 100644
--- a/docs/articles/vignette_visualization.html
+++ b/docs/articles/vignette_visualization.html
@@ -112,7 +112,7 @@
     
    
       
    Visualization
    
             
-            
    2023-11-07
    
+            
    2023-11-10
    
       
       Source: vignettes/vignette_visualization.Rmd
       
diff --git a/docs/index.html b/docs/index.html
index 00d82b68..850aaeef 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -119,7 +119,7 @@ 
    
 
    Summary
 
    
-
    The gcamfaostat R package is designed for the preparation, processing, and synthesis of the Food and Agriculture Organization (FAO) Statistics (FAOSTAT) agroeconomic data. The primary purpose is to facilitate FAOSTAT data use in global economic and multisector dynamic models while ensuring transparency, traceability, and reproducibility. Here, we provide an overview of the development of gcamfaostat (v1.0) and demonstrate its capabilities in generating and maintaining agroeconomic data required for the Global Change Analysis Model (GCAM). Our initiative seeks to enhance the quality and accessibility of data for the global agroeconomic modeling community, with the aim of fostering more robust and harmonized outcomes in a collaborative, efficient, and open-source framework. One of the important features of the package is the possibility to construct the FAO Food Balance Sheets at the disaggregated commodity level (with over 500 commodities), which provides a comprehensive and detailed data input for a variety of analytical and modeling applications. The processed data and visualizations offered by gcamfaostat can also be valuable to a broader audience interested in gaining insights into the intricacies of global agriculture.
    
+
    The gcamfaostat R package is designed for the preparation, processing, and synthesis of the Food and Agriculture Organization (FAO) Statistics (FAOSTAT) agroeconomic data. The primary purpose is to facilitate FAOSTAT data use in global economic and multisector dynamic models while ensuring transparency, traceability, and reproducibility. Here, we provide an overview of the development of gcamfaostat (v1.0.0) and demonstrate its capabilities in generating and maintaining agroeconomic data required for the Global Change Analysis Model (GCAM). Our initiative seeks to enhance the quality and accessibility of data for the global agroeconomic modeling community, with the aim of fostering more robust and harmonized outcomes in a collaborative, efficient, and open-source framework. One of the important features of the package is the possibility to construct the FAO Food Balance Sheets at the disaggregated commodity level (with over 500 commodities), which provides a comprehensive and detailed data input for a variety of analytical and modeling applications. The processed data and visualizations offered by gcamfaostat can also be valuable to a broader audience interested in gaining insights into the intricacies of global agriculture.
    
 
    This tool bridges a crucial gap in the literature by offering several key features and capabilities.
    
 
      
 
    1. 
@@ -129,7 +129,7 @@ 
      SummaryExpandability and Consistency: gcamfaostat can be used to flexibly process and update agroeconomic data for any agroeconomic model. The package framework can be also easily expanded to include new modules for consistently processing new data.
      
 
      2. 
 
    2. 
-Community Collaboration and Efficiency: The package provides an open-source platform for researchers to continually enhance the processing methods. This collaborative approach, which establishes a standardized and streamlined process for data preparation and processing, carries benefits that extend to all modeling groups. By reducing the efforts required for data processing and fostering harmonized base data calibration, it contributes to a reduction in modeling uncertainty and enhances the overall research efficiency.
      
+Community Collaboration and Efficiency: The package provides an open-source platform for researchers to continually enhance the processing methods. This collaborative approach, which establishes a standardized and streamlined process for data preparation and processing, carries benefits that extend to all modeling groups. By reducing the effort required for data processing and fostering harmonized base data calibration, it contributes to a reduction in modeling uncertainty and enhances the overall research efficiency.
      
 
      3. 
 
    3. 
 User Accessibility: Where applicable, the processed data can be mapped and aggregated to user-specified regions and sectors for agroeconomic modeling. However, beyond the modeling community, gcamfaostat can be valuable to a broader range of users interested in understanding global agriculture trends and dynamics, as it provides user-friendly data processing and visualization tools.
      4. 
@@ -232,7 +232,7 @@ 
      Future work and contributionRelated publicationshttps://doi.org/10.5281/zenodo.8260225
 
 

-
Copyright 2019 Battelle Memorial Institute; see the LICENSE file.

+
Copyright 2023 Battelle Memorial Institute; see the LICENSE file.
diff --git a/docs/paper.html b/docs/paper.html index 7cfcb80f..08e0522b 100644 --- a/docs/paper.html +++ b/docs/paper.html @@ -91,17 +91,17 @@

Summary

-

The gcamfaostat R package is designed for the preparation, processing, and synthesis of the Food and Agriculture Organization (FAO) Statistics (FAOSTAT) agroeconomic data. The primary purpose is to facilitate FAOSTAT data use in global economic and multisector dynamic models while ensuring transparency, traceability, and reproducibility. Here, we provide an overview of the development of gcamfaostat v1.0 and demonstrate its capabilities in generating and maintaining agroeconomic data required for the Global Change Analysis Model (GCAM). Our initiative seeks to enhance the quality and accessibility of data for the global agroeconomic modeling community, with the aim of fostering more robust and harmonized outcomes in a collaborative, efficient, and open-source framework. One of the important features of the package is the possibility to construct the FAO Food Balance Sheets at the disaggregated commodity level (with over 500 commodities), which provides a comprehensive and detailed data input for a variety of analytical and modeling applications. The processed data and visualizations offered by gcamfaostat can also be valuable to a broader audience interested in gaining insights into the intricacies of global agriculture.

+

The gcamfaostat R package is designed for the preparation, processing, and synthesis of the Food and Agriculture Organization (FAO) Statistics (FAOSTAT) agroeconomic data. The primary purpose is to facilitate FAOSTAT data use in global economic and multisector dynamic models while ensuring transparency, traceability, and reproducibility. Here, we provide an overview of the development of gcamfaostat v1.0.0 and demonstrate its capabilities in generating and maintaining agroeconomic data required for the Global Change Analysis Model (GCAM). Our initiative seeks to enhance the quality and accessibility of data for the global agroeconomic modeling community, with the aim of fostering more robust and harmonized outcomes in a collaborative, efficient, and open-source framework. One of the important features of the package is the possibility to construct the FAO Food Balance Sheets at the disaggregated commodity level (with over 500 commodities), which provides a comprehensive and detailed data input for a variety of analytical and modeling applications. The processed data and visualizations offered by gcamfaostat can also be valuable to a broader audience interested in gaining insights into the intricacies of global agriculture.

Statement of need

-

Global economic and multisector dynamic models have become pivotal tools for investigating complex interactions between human activities and the environment, as evident in recent research [@Doelman2022Quantifying;@Fujimori2022Land-based;@IPCC2022Annex;@Ven2023multimodel]. Agriculture and land use (AgLU) plays a critical role in these models, particularly when used to address key agroeconomic questions [@Graham2023Agricultural;@Yarlagadda2023Trade;@Zhang2023Agriculture;@Zhao2021Global;@Zhao2020critical]. Sound economic modeling hinges significantly upon the accessibility and quality of data [@Bruckner2019FABIO;@Calvin2022GMD;@Chepeliev2022JGEA]. The FAOSTAT serves as one of the key global data sources, offering open-access data on country-level agricultural production, land use, trade, food consumption, nutrient content, prices, and more [@FAOSTAT2023FAOSTAT]. However, the raw data from FAOSTAT requires cleaning, balancing, and synthesis, involving assumptions such as interpolation and mapping, which can introduce uncertainties. In addition, some of the core datasets reported by FAOSTAT, such as FAO’s Food Balance Sheets (FBS), are compiled at a specific level of aggregation, combining together primary and processed commodities (e.g., wheat and flour), which creates additional data processing challenges for the agroeconomic modeling community [@Chepeliev2022JGEA]. It is noteworthy that each agroeconomic modeling team typically develops its own assumptions and methods to prepare and process FAOSTAT data [@bond2019gcamdata]. While largely overlooked, the uncertainty in the base data calibration approach likely contributed to the disparities in model outcomes [@Lampe2014AgMIP;@zhao2021role]. Hence, our motivation is to create an open-source tool (gcamfaostat) for the preparation, processing, and synthesis of FAOSTAT data for global agroeconomic modeling. This tool bridges a crucial gap in the literature by offering several key features and capabilities.

+

Global economic and multisector dynamic models have become pivotal tools for investigating complex interactions between human activities and the environment, as evident in recent research [@Doelman2022Quantifying;@Fujimori2022Land-based;@IPCC2022Annex;@Ven2023multimodel]. Agriculture and land use (AgLU) plays a critical role in these models, particularly when used to address key agroeconomic questions [@Graham2023Agricultural;@Yarlagadda2023Trade;@Zhang2023Agriculture;@Zhao2021Global;@Zhao2020critical]. Sound economic modeling hinges significantly upon the accessibility and quality of data [@Bruckner2019FABIO;@Calvin2022GMD;@Chepeliev2022JGEA]. The FAOSTAT serves as one of the key global data sources, offering open-access data on country-level agricultural production, land use, trade, food consumption, nutrient content, prices, and more [@FAOSTAT2023FAOSTAT]. However, the raw data from FAOSTAT requires cleaning, balancing, and synthesis, involving assumptions such as interpolation and mapping, which can introduce uncertainties. In addition, some of the core datasets reported by FAOSTAT, such as FAO’s Food Balance Sheets (FBS), are compiled at a specific level of aggregation, combining together primary and processed commodities (e.g., wheat and flour), which creates additional data processing challenges for the agroeconomic modeling community [@Chepeliev2022JGEA]. It is noteworthy that each agroeconomic modeling team typically develops its own assumptions and methods to prepare and process FAOSTAT data [@bond2019gcamdata]. While largely overlooked, the uncertainty in the base data calibration approach likely contribute to the disparities in model outcomes [@Lampe2014AgMIP;@zhao2021role]. Hence, our motivation is to create an open-source tool (gcamfaostat) for the preparation, processing, and synthesis of FAOSTAT data for global agroeconomic modeling. This tool bridges a crucial gap in the literature by offering several key features and capabilities.

  1. Transparency and Reproducibility: gcamfaostat incorporates functions for downloading, cleaning, synthesizing, and balancing agroeconomic datasets in a traceable, transparent, and reproducible manner. This enhances the credibility of the processing and allows for better scrutiny of the methods. We have documented and demonstrated the use of the package in generating and updating agroeconomic data needed for the GCAM.
  2. Expandability and Consistency: gcamfaostat can be used to flexibly process and update agroeconomic data for any agroeconomic model. The package framework can be also easily expanded to include new modules for consistently processing new data.
  3. -Community Collaboration and Efficiency: The package provides an open-source platform for researchers to continually enhance the processing methods. This collaborative approach, which establishes a standardized and streamlined process for data preparation and processing, carries benefits that extend to all modeling groups. By reducing the efforts required for data processing and fostering harmonized base data calibration, it contributes to a reduction in modeling uncertainty and enhances the overall research efficiency.
    4. +Community Collaboration and Efficiency: The package provides an open-source platform for researchers to continually enhance the processing methods. This collaborative approach, which establishes a standardized and streamlined process for data preparation and processing, carries benefits that extend to all modeling groups. By reducing the effort required for data processing and fostering harmonized base data calibration, it contributes to a reduction in modeling uncertainty and enhances the overall research efficiency.
  4. User Accessibility: Where applicable, the processed data can be mapped and aggregated to user-specified regions and sectors for agroeconomic modeling. However, beyond the modeling community, gcamfaostat can be valuable to a broader range of users interested in understanding global agriculture trends and dynamics, as it provides user-friendly data processing and visualization tools.
@@ -109,8 +109,8 @@

Statement of needDesign and Functionality

Bridging the gap between FAOSTAT and global economic modeling

-

shows a standard framework of using FAOSTAT data in GCAM. GCAM is a widely recognized global economic and multisector dynamic model complemented by the gcamdata R package, which serves as its data processing system. Particularly, gcamdata includes modules (data processing chunks) and functions to convert raw data inputs into hundreds of XML files needed by GCAM [@bond2019gcamdata]. As an illustration, in the latest GCAM version, GCAM v7 [@bond_lamberty_2023], about 280 XML files, with a combined size of 4.1 GB, are generated. Although AgLU-related XMLs represent only about 10% of the total number of files, they contribute over 50% in size (~2.1 GB). The majority of AgLU-related data, whether directly or indirectly, rely on raw data sourced from the FAOSTAT.

-

Nonetheless, the FAOSTAT data employed within gcamdata has traditionally involved manual downloads and may have undergone preprocessing. In light of the increasing data needs, maintaining the FAOSTAT data processing tasks in gcamdata has become increasingly challenging. In addition, the processing of FAOSTAT data in the AgLU modules of gcamdata is tailored specifically for GCAM. Consequently, the integration of FAOSTAT data updates has proven to be a non-trivial task, and the data processed by the AgLU module has limited applicability in other modeling contexts [@zhao_cmp360]. The gcamfaostat package is aimed to address these limitations (). The targeted approach incorporates data preparation, processing, and synthesizing capabilities within a dedicated package, gcamfaostat, while regional and sectoral aggregation functions in the model data system are implemented using standalone routines within the gcamdata package. This strategy not only ensures the streamlined operation of gcamfaostat but also contributes to keeping model data system lightweight and more straightforward to maintain.

+

shows a standard framework of using FAOSTAT data in GCAM. GCAM is a widely recognized global economic and multisector dynamic model complemented by the gcamdata R package, which serves as its data processing system. Particularly, gcamdata includes modules (data processing chunks) and functions to convert raw data inputs into hundreds of XML input files used by GCAM [@bond2019gcamdata]. As an illustration, in the latest GCAM version, GCAM v7 [@bond_lamberty_2023], about 280 XML files, with a combined size of 4.1 GB, are generated. Although AgLU-related XMLs represent only about 10% of the total number of files, they contribute over 50% in size (~2.1 GB). The majority of AgLU-related data, whether directly or indirectly, rely on raw data sourced from the FAOSTAT.

+

Nonetheless, the FAOSTAT data employed within gcamdata has traditionally involved manual downloads and may have undergone preprocessing. In light of the increasing data needs, maintaining the FAOSTAT data processing tasks in gcamdata has become increasingly challenging. In addition, the processing of FAOSTAT data in the AgLU modules of gcamdata is tailored specifically for GCAM. Consequently, the integration of FAOSTAT data updates has proven to be a non-trivial task, and the data processed by the AgLU module has limited applicability in other modeling contexts [@zhao_cmp360]. The gcamfaostat package aims to address these limitations (). The targeted approach incorporates data preparation, processing, and synthesis capabilities within a dedicated package, gcamfaostat, while regional and sectoral aggregation functions in the model data system are implemented using standalone routines within the gcamdata package. This strategy not only ensures the streamlined operation of gcamfaostat but also contributes to keeping model data system lightweight and more straightforward to maintain.

The original framework of utilizing FAOSTAT data in GCAM and similar large-scale models. Note that FAOSTAT data is mainly processed in the AgLU modules in gcamdata while there could be interdependency across data processing modules.
The original framework of utilizing FAOSTAT data in GCAM and similar large-scale models. Note that FAOSTAT data is mainly processed in the AgLU modules in gcamdata while there could be interdependency across data processing modules.
@@ -138,7 +138,7 @@

Data preparationThe archived data is consistent with the Prebuilt package data.

FF_download_FAOSTAT()

  • The function downloads the latest raw data from FAOSTAT.
    • -

Table 1. FAOSTAT dataset processed in gcamfaostat v1.0.

+

Table 1. FAOSTAT dataset processed in gcamfaostat v1.0.0.

@@ -232,7 +232,7 @@

Data processing
The architecture of data processing modules in gcamfaostat.

Data synthesizing in a key module

-

Of particular significance is the module_xfaostat_L105_DataConnectionToSUA, which plays a pivotal role in harmonizing various FAOSTAT datasets to generate a cohesive set of agricultural supply and utilization accounts (SUA) data. This complex process is elucidated in . This endeavor entails working through nine tiers of data, each sourced differently, with the aim of producing and harmonizing agricultural SUA dataset for a diverse range of over 500 agricultural commodities. Compared to the FAO’s FBS, which report food and nutritional information for about 100 composite categories, in many cases combining primary and processed commodities (e.g., wheat and flour) within a single category (e.g., wheat and products), the constructed SUA explicitly trace the transformations between primary and processed commodities, while reporting nutritional details at the highly disaggregated level (over 500 commodities). In doing so, the constructed dataset substantially simplifies the FAOSTAT data processing steps by explicitly distinguishing food and nutritional supply at the individual commodity level and facilitating a straightforward mapping of the corresponding data to the global agroeconomic models.

+

Of particular significance is the module_xfaostat_L105_DataConnectionToSUA, which plays a pivotal role in harmonizing various FAOSTAT datasets to generate a cohesive set of agricultural supply and utilization accounts (SUA) data. This complex process is elucidated in . This endeavor entails working through nine tiers of data, each sourced differently, with the aim of producing an harmonized agricultural SUA dataset for over 500 agricultural commodities. Compared to the FAO’s FBS, which report food and nutritional information for about 100 composite categories, in many cases combining primary and processed commodities (e.g., wheat and flour) within a single category (e.g., wheat and products), the constructed SUA explicitly trace the transformations between primary and processed commodities, while reporting nutritional details at a highly disaggregated level (over 500 commodities). In doing so, the constructed dataset substantially simplifies the FAOSTAT data processing steps by explicitly distinguishing food and nutritional supply at the individual commodity level and facilitating a straightforward mapping of the corresponding data to the global agroeconomic models.

As an illustrative example, the first tier comprises 168 commodities, generated by combining production data from QCL, trade data from TM, and other essential balancing elements (such as opening and closing stocks, food and feed uses, and other industrial uses) from SCL. For a more comprehensive understanding of these procedures, we encourage an interested user to explore the mapping file, FAO_items. It is crucial to underscore the importance of these processing procedures, as raw FAOSTAT data often contains duplicated elements and inconsistencies among different datasets. For instance, trade data can be found in TCL, TM, SCL, and FBS, while production data exists in QCL and SCL (please see Table 1 for the corresponding dataset codes).

FAOSTAT agricultural supply utilization data synthesis in module_xfaostat_L105_DataConnectionToSUA. Note that the nine tiers of data, distinguished by commodities (or items in FAOSTAT terms) included, have different sources for generating agricultural supply utilization accounts.
FAOSTAT agricultural supply utilization data synthesis in module_xfaostat_L105_DataConnectionToSUA. Note that the nine tiers of data, distinguished by commodities (or items in FAOSTAT terms) included, have different sources for generating agricultural supply utilization accounts.
@@ -259,13 +259,13 @@

Data tracingVisualization

In addition to generating data for modeling purposes, we also provide illustrative examples for visualizing the key data elements. Here are some examples for using the processed data to illustrate the connection of harvested area () and food calories () via supply utilization accounts (). These figures focus on the 2013 – 2017 mean values as they are the base calibration years of GCAM. A user can change the years and mappings as desired.

-World area harvested (shares) grouped by GCAM crops based on the 2013 – 2017 mean values. The total harvested area is 1509 million hectares (Mha).
World area harvested (shares) grouped by GCAM crops based on the 2013 – 2017 mean values. The total harvested area is 1509 million hectares (Mha).
+World area harvested (shares) grouped by GCAM crops based on the 2013 – 2017 mean values. The total harvested area is 1509 million hectares (Mha).
World area harvested (shares) grouped by GCAM crops based on the 2013 – 2017 mean values. The total harvested area is 1509 million hectares (Mha).
-Food calories availability per capita per day grouped by GCAM regions and commodities based on the 2013 – 2017 mean values. The world average value is 2902 Kcal per capita per day (Kcal/ca/d).
Food calories availability per capita per day grouped by GCAM regions and commodities based on the 2013 – 2017 mean values. The world average value is 2902 Kcal per capita per day (Kcal/ca/d).
+Food calories availability per capita per day grouped by GCAM regions and commodities based on the 2013 – 2017 mean values. The world average value is 2902 Kcal per capita per day (Kcal/ca/d).
Food calories availability per capita per day grouped by GCAM regions and commodities based on the 2013 – 2017 mean values. The world average value is 2902 Kcal per capita per day (Kcal/ca/d).
-World supply utilization accounts for key commodities based on the 2013 – 2017 mean values. Note that negative values are used for demand categories and positive values are used for supply categories. Other uses include seed use, losses, and industrial use; residuals (mostly small) indicate the imbalance in the data. The total supply is equal to the total demand.
World supply utilization accounts for key commodities based on the 2013 – 2017 mean values. Note that negative values are used for demand categories and positive values are used for supply categories. Other uses include seed use, losses, and industrial use; residuals (mostly small) indicate the imbalance in the data. The total supply is equal to the total demand.
+World supply utilization accounts for key commodities based on the 2013 – 2017 mean values. Note that negative values are used for demand categories and positive values are used for supply categories. Other uses include seed use, losses, and industrial use; residuals (mostly small) indicate the imbalance in the data. The total supply is equal to the total demand.
World supply utilization accounts for key commodities based on the 2013 – 2017 mean values. Note that negative values are used for demand categories and positive values are used for supply categories. Other uses include seed use, losses, and industrial use; residuals (mostly small) indicate the imbalance in the data. The total supply is equal to the total demand.
@@ -303,14 +303,14 @@

Future work and contribution

-

This research was supported by the US Department of Energy, Office of Science, as part of research in MultiSector Dynamics, Earth and Environmental System Modeling Program. The Pacific Northwest National Laboratory is operated for DOE by Battelle Memorial Institute under contract DE-AC05-76RL01830. Dr. Calvin is currently detailed to the National Aeronautics and Space Administration. Dr. Calvin’s contributions to this article occurred prior to her detail. The views expressed are her own and do not necessarily represent the views of the National Aeronautics and Space Administration or the United States Government.

+

This research was supported by the US Department of Energy, Office of Science, as part of research in MultiSector Dynamics, Earth and Environmental System Modeling Program. The Pacific Northwest National Laboratory is operated for DOE by Battelle Memorial Institute under contract DE-AC05-76RL01830. Dr. Calvin is currently detailed to the National Aeronautics and Space Administration. Dr. Calvin’s contributions to this article occurred prior to her detail. The views expressed are her own and do not necessarily represent the views of the National Aeronautics and Space Administration or the United States Government. We extend our sincere appreciation to Matthew Binsted and Page Kyle for their invaluable contributions.

References

diff --git a/docs/pkgdown.yml b/docs/pkgdown.yml index 7daf3cb0..697c7a93 100644 --- a/docs/pkgdown.yml +++ b/docs/pkgdown.yml @@ -8,7 +8,7 @@ articles: vignette_processing_flow: vignette_processing_flow.html vignette_use_cases: vignette_use_cases.html vignette_visualization: vignette_visualization.html -last_built: 2023-11-07T06:27Z +last_built: 2023-11-11T03:08Z urls: reference: https://jgcri.github.io/gcamfaostat/reference article: https://jgcri.github.io/gcamfaostat/articles diff --git a/docs/reference/FAOSTAT_metadata.html b/docs/reference/FAOSTAT_metadata.html index 86ea2695..cd97ce14 100644 --- a/docs/reference/FAOSTAT_metadata.html +++ b/docs/reference/FAOSTAT_metadata.html @@ -126,7 +126,7 @@

Examples

#> } #> return(metadata) #> } -#> <bytecode: 0x000000002e129960> +#> <bytecode: 0x000000002cfcd4c8> #> <environment: namespace:gcamfaostat>
diff --git a/inst/extdata/aglu/FAO/FAOSTAT/metadata_log/FAOSTAT_METADATA_2023-11-10.csv b/inst/extdata/aglu/FAO/FAOSTAT/metadata_log/FAOSTAT_METADATA_2023-11-10.csv new file mode 100644 index 00000000..f6896b4a --- /dev/null +++ b/inst/extdata/aglu/FAO/FAOSTAT/metadata_log/FAOSTAT_METADATA_2023-11-10.csv @@ -0,0 +1,14 @@ +datasetcode,datasetname,topic,datasetdescription,contact,email,dateupdate,compressionformat,filetype,filesize,filerows,filelocation +CB,Food Balances: Commodity Balances (non-food) (2010-),"Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2023-08-08T00:00:00,zip,csv,859KB,117560,https://fenixservices.fao.org/faostat/static/bulkdownloads/CommodityBalances_(non-food)_E_All_Data_(Normalized).zip +FBS,Food Balances: Food Balances (2010-),"Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2023-09-04T00:00:00,zip,csv,55894KB,4320908,https://fenixservices.fao.org/faostat/static/bulkdownloads/FoodBalanceSheets_E_All_Data_(Normalized).zip +FBSH,"Food Balances: Food Balances (-2013, old methodology and population)","Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2023-03-10T00:00:00,zip,csv,70922KB,11479903,https://fenixservices.fao.org/faostat/static/bulkdownloads/FoodBalanceSheetsHistoric_E_All_Data_(Normalized).zip +FO,Forestry: Forestry Production and Trade,"Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2023-09-05T00:00:00,zip,csv,15739KB,2237225,https://fenixservices.fao.org/faostat/static/bulkdownloads/Forestry_E_All_Data_(Normalized).zip +OA,Population and Employment: Annual population,"Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations,,,2022-10-24T00:00:00,zip,csv,1540KB,169142,https://fenixservices.fao.org/faostat/static/bulkdownloads/Population_E_All_Data_(Normalized).zip +PD,Prices: Deflators,FAO viale delle terme di Caracalla 1,Food and Agriculture Organization of the United Nations (FAO),,,2023-08-16T00:00:00,zip,csv,905KB,83486,https://fenixservices.fao.org/faostat/static/bulkdownloads/Deflators_E_All_Data_(Normalized).zip +PP,Prices: Producer Prices,"Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2023-02-23T00:00:00,zip,csv,9911KB,1150071,https://fenixservices.fao.org/faostat/static/bulkdownloads/Prices_E_All_Data_(Normalized).zip +QCL,Production: Crops and livestock products,"Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2023-03-22T00:00:00,zip,csv,30139KB,3761168,https://fenixservices.fao.org/faostat/static/bulkdownloads/Production_Crops_Livestock_E_All_Data_(Normalized).zip +RFN,"Land, Inputs and Sustainability: Fertilizers by Nutrient",,Food and Agriculture Organization of the United Nations (FAO),,,2023-07-05T00:00:00,zip,csv,1796KB,220984,https://fenixservices.fao.org/faostat/static/bulkdownloads/Inputs_FertilizersNutrient_E_All_Data_(Normalized).zip +RL,"Land, Inputs and Sustainability: Land Use","Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2023-07-10T00:00:00,zip,csv,2538KB,376295,https://fenixservices.fao.org/faostat/static/bulkdownloads/Inputs_LandUse_E_All_Data_(Normalized).zip +SCL,Food Balances: Supply Utilization Accounts (2010-),"Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2023-09-04T00:00:00,zip,csv,73725KB,9959324,https://fenixservices.fao.org/faostat/static/bulkdownloads/SUA_Crops_Livestock_E_All_Data_(Normalized).zip +TCL,Trade: Crops and livestock products,"Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2023-08-14T00:00:00,zip,csv,234036KB,15697216,https://fenixservices.fao.org/faostat/static/bulkdownloads/Trade_CropsLivestock_E_All_Data_(Normalized).zip +TM,Trade: Detailed trade matrix,"Viale delle Terme di Caracalla, 00153 Rome, Italy",Food and Agriculture Organization of the United Nations (FAO),,,2022-02-14T00:00:00,zip,csv,465008KB,46807399,https://fenixservices.fao.org/faostat/static/bulkdownloads/Trade_DetailedTradeMatrix_E_All_Data_(Normalized).zip diff --git a/paper.md b/paper.md index 6343e2c7..2a77f1a4 100644 --- a/paper.md +++ b/paper.md @@ -39,15 +39,15 @@ affiliations: # Summary -The **`gcamfaostat`** R package is designed for the preparation, processing, and synthesis of the Food and Agriculture Organization (FAO) Statistics ([FAOSTAT](https://www.fao.org/faostat/en/#data)) agroeconomic data. The primary purpose is to facilitate FAOSTAT data use in global economic and multisector dynamic models while ensuring transparency, traceability, and reproducibility. Here, we provide an overview of the development of **`gcamfaostat v1.0`** and demonstrate its capabilities in generating and maintaining agroeconomic data required for the Global Change Analysis Model ([GCAM](https://jgcri.github.io/gcam-doc/toc.html)). Our initiative seeks to enhance the quality and accessibility of data for the global agroeconomic modeling community, with the aim of fostering more robust and harmonized outcomes in a collaborative, efficient, and open-source framework. One of the important features of the package is the possibility to construct the FAO Food Balance Sheets at the disaggregated commodity level (with over 500 commodities), which provides a comprehensive and detailed data input for a variety of analytical and modeling applications. The processed data and visualizations offered by **`gcamfaostat`** can also be valuable to a broader audience interested in gaining insights into the intricacies of global agriculture. +The **`gcamfaostat`** R package is designed for the preparation, processing, and synthesis of the Food and Agriculture Organization (FAO) Statistics ([FAOSTAT](https://www.fao.org/faostat/en/#data)) agroeconomic data. The primary purpose is to facilitate FAOSTAT data use in global economic and multisector dynamic models while ensuring transparency, traceability, and reproducibility. Here, we provide an overview of the development of **`gcamfaostat v1.0.0`** and demonstrate its capabilities in generating and maintaining agroeconomic data required for the Global Change Analysis Model ([GCAM](https://jgcri.github.io/gcam-doc/toc.html)). Our initiative seeks to enhance the quality and accessibility of data for the global agroeconomic modeling community, with the aim of fostering more robust and harmonized outcomes in a collaborative, efficient, and open-source framework. One of the important features of the package is the possibility to construct the FAO Food Balance Sheets at the disaggregated commodity level (with over 500 commodities), which provides a comprehensive and detailed data input for a variety of analytical and modeling applications. The processed data and visualizations offered by **`gcamfaostat`** can also be valuable to a broader audience interested in gaining insights into the intricacies of global agriculture. # Statement of need -Global economic and multisector dynamic models have become pivotal tools for investigating complex interactions between human activities and the environment, as evident in recent research [@Doelman2022Quantifying;@Fujimori2022Land-based;@IPCC2022Annex;@Ven2023multimodel]. Agriculture and land use (AgLU) plays a critical role in these models, particularly when used to address key agroeconomic questions [@Graham2023Agricultural;@Yarlagadda2023Trade;@Zhang2023Agriculture;@Zhao2021Global;@Zhao2020critical]. Sound economic modeling hinges significantly upon the accessibility and quality of data [@Bruckner2019FABIO;@Calvin2022GMD;@Chepeliev2022JGEA]. The FAOSTAT serves as one of the key global data sources, offering open-access data on country-level agricultural production, land use, trade, food consumption, nutrient content, prices, and more [@FAOSTAT2023FAOSTAT]. However, the raw data from FAOSTAT requires cleaning, balancing, and synthesis, involving assumptions such as interpolation and mapping, which can introduce uncertainties. In addition, some of the core datasets reported by FAOSTAT, such as FAO’s Food Balance Sheets (FBS), are compiled at a specific level of aggregation, combining together primary and processed commodities (e.g., wheat and flour), which creates additional data processing challenges for the agroeconomic modeling community [@Chepeliev2022JGEA]. It is noteworthy that each agroeconomic modeling team typically develops its own assumptions and methods to prepare and process FAOSTAT data [@bond2019gcamdata]. While largely overlooked, the uncertainty in the base data calibration approach likely contributed to the disparities in model outcomes [@Lampe2014AgMIP;@zhao2021role]. Hence, our motivation is to create an open-source tool (**`gcamfaostat`**) for the preparation, processing, and synthesis of FAOSTAT data for global agroeconomic modeling. This tool bridges a crucial gap in the literature by offering several key features and capabilities. +Global economic and multisector dynamic models have become pivotal tools for investigating complex interactions between human activities and the environment, as evident in recent research [@Doelman2022Quantifying;@Fujimori2022Land-based;@IPCC2022Annex;@Ven2023multimodel]. Agriculture and land use (AgLU) plays a critical role in these models, particularly when used to address key agroeconomic questions [@Graham2023Agricultural;@Yarlagadda2023Trade;@Zhang2023Agriculture;@Zhao2021Global;@Zhao2020critical]. Sound economic modeling hinges significantly upon the accessibility and quality of data [@Bruckner2019FABIO;@Calvin2022GMD;@Chepeliev2022JGEA]. The FAOSTAT serves as one of the key global data sources, offering open-access data on country-level agricultural production, land use, trade, food consumption, nutrient content, prices, and more [@FAOSTAT2023FAOSTAT]. However, the raw data from FAOSTAT requires cleaning, balancing, and synthesis, involving assumptions such as interpolation and mapping, which can introduce uncertainties. In addition, some of the core datasets reported by FAOSTAT, such as FAO’s Food Balance Sheets (FBS), are compiled at a specific level of aggregation, combining together primary and processed commodities (e.g., wheat and flour), which creates additional data processing challenges for the agroeconomic modeling community [@Chepeliev2022JGEA]. It is noteworthy that each agroeconomic modeling team typically develops its own assumptions and methods to prepare and process FAOSTAT data [@bond2019gcamdata]. While largely overlooked, the uncertainty in the base data calibration approach likely contribute to the disparities in model outcomes [@Lampe2014AgMIP;@zhao2021role]. Hence, our motivation is to create an open-source tool (**`gcamfaostat`**) for the preparation, processing, and synthesis of FAOSTAT data for global agroeconomic modeling. This tool bridges a crucial gap in the literature by offering several key features and capabilities. 1. **Transparency and Reproducibility**: **`gcamfaostat`** incorporates functions for downloading, cleaning, synthesizing, and balancing agroeconomic datasets in a traceable, transparent, and reproducible manner. This enhances the credibility of the processing and allows for better scrutiny of the methods. We have documented and demonstrated the use of the package in generating and updating agroeconomic data needed for the GCAM. 2. **Expandability and Consistency**: **`gcamfaostat`** can be used to flexibly process and update agroeconomic data for any agroeconomic model. The package framework can be also easily expanded to include new modules for consistently processing new data. -3. **Community Collaboration and Efficiency**: The package provides an open-source platform for researchers to continually enhance the processing methods. This collaborative approach, which establishes a standardized and streamlined process for data preparation and processing, carries benefits that extend to all modeling groups. By reducing the efforts required for data processing and fostering harmonized base data calibration, it contributes to a reduction in modeling uncertainty and enhances the overall research efficiency. +3. **Community Collaboration and Efficiency**: The package provides an open-source platform for researchers to continually enhance the processing methods. This collaborative approach, which establishes a standardized and streamlined process for data preparation and processing, carries benefits that extend to all modeling groups. By reducing the effort required for data processing and fostering harmonized base data calibration, it contributes to a reduction in modeling uncertainty and enhances the overall research efficiency. 4. **User Accessibility**: Where applicable, the processed data can be mapped and aggregated to user-specified regions and sectors for agroeconomic modeling. However, beyond the modeling community, **`gcamfaostat`** can be valuable to a broader range of users interested in understanding global agriculture trends and dynamics, as it provides user-friendly data processing and visualization tools. # Design and Functionality @@ -55,9 +55,9 @@ Global economic and multisector dynamic models have become pivotal tools for inv ## Bridging the gap between FAOSTAT and global economic modeling -\autoref{fig:Fig1} shows a standard framework of using FAOSTAT data in GCAM. GCAM is a widely recognized global economic and multisector dynamic model complemented by the gcamdata R package, which serves as its data processing system. Particularly, gcamdata includes modules (data processing chunks) and functions to convert raw data inputs into hundreds of XML files needed by GCAM [@bond2019gcamdata]. As an illustration, in the latest GCAM version, GCAM v7 [@bond_lamberty_2023], about 280 XML files, with a combined size of 4.1 GB, are generated. Although AgLU-related XMLs represent only about 10% of the total number of files, they contribute over 50% in size (~2.1 GB). The majority of AgLU-related data, whether directly or indirectly, rely on raw data sourced from the FAOSTAT. +\autoref{fig:Fig1} shows a standard framework of using FAOSTAT data in GCAM. GCAM is a widely recognized global economic and multisector dynamic model complemented by the gcamdata R package, which serves as its data processing system. Particularly, gcamdata includes modules (data processing chunks) and functions to convert raw data inputs into hundreds of XML input files used by GCAM [@bond2019gcamdata]. As an illustration, in the latest GCAM version, GCAM v7 [@bond_lamberty_2023], about 280 XML files, with a combined size of 4.1 GB, are generated. Although AgLU-related XMLs represent only about 10% of the total number of files, they contribute over 50% in size (~2.1 GB). The majority of AgLU-related data, whether directly or indirectly, rely on raw data sourced from the FAOSTAT. -Nonetheless, the FAOSTAT data employed within gcamdata has traditionally involved manual downloads and may have undergone preprocessing. In light of the increasing data needs, maintaining the FAOSTAT data processing tasks in gcamdata has become increasingly challenging. In addition, the processing of FAOSTAT data in the AgLU modules of gcamdata is tailored specifically for GCAM. Consequently, the integration of FAOSTAT data updates has proven to be a non-trivial task, and the data processed by the AgLU module has limited applicability in other modeling contexts [@zhao_cmp360]. The **`gcamfaostat`** package is aimed to address these limitations (\autoref{fig:Fig2}). The targeted approach incorporates data preparation, processing, and synthesizing capabilities within a dedicated package, gcamfaostat, while regional and sectoral aggregation functions in the model data system are implemented using standalone routines within the gcamdata package. This strategy not only ensures the streamlined operation of **`gcamfaostat`** but also contributes to keeping model data system lightweight and more straightforward to maintain. +Nonetheless, the FAOSTAT data employed within gcamdata has traditionally involved manual downloads and may have undergone preprocessing. In light of the increasing data needs, maintaining the FAOSTAT data processing tasks in gcamdata has become increasingly challenging. In addition, the processing of FAOSTAT data in the AgLU modules of gcamdata is tailored specifically for GCAM. Consequently, the integration of FAOSTAT data updates has proven to be a non-trivial task, and the data processed by the AgLU module has limited applicability in other modeling contexts [@zhao_cmp360]. The **`gcamfaostat`** package aims to address these limitations (\autoref{fig:Fig2}). The targeted approach incorporates data preparation, processing, and synthesis capabilities within a dedicated package, gcamfaostat, while regional and sectoral aggregation functions in the model data system are implemented using standalone routines within the gcamdata package. This strategy not only ensures the streamlined operation of **`gcamfaostat`** but also contributes to keeping model data system lightweight and more straightforward to maintain. ![The original framework of utilizing FAOSTAT data in GCAM and similar large-scale models. Note that FAOSTAT data is mainly processed in the AgLU modules in gcamdata while there could be interdependency across data processing modules. \label{fig:Fig1}](./man/figures/Fig_FAOSTAT_gcamdata.jpg){width=70%} @@ -93,7 +93,7 @@ In this section we describe key functions included in gcamfaostat. More details * The function downloads the latest raw data from FAOSTAT. -Table 1. FAOSTAT dataset processed in gcamfaostat v1.0. +Table 1. FAOSTAT dataset processed in gcamfaostat v1.0.0. | Dataset Code | Dataset Name | Exist_Local | Exist_Prebuilt | FAO update date | FAO size | |:------------:|:----------------------------------------------------------:|:-----------:|:--------------:|:--------------:|:--------:| @@ -123,7 +123,7 @@ The architecture of gcamfaostat processing modules is depicted in \autoref{fig:F **Data synthesizing in a key module** -Of particular significance is the `module_xfaostat_L105_DataConnectionToSUA`, which plays a pivotal role in harmonizing various FAOSTAT datasets to generate a cohesive set of agricultural supply and utilization accounts (SUA) data. This complex process is elucidated in \autoref{fig:Fig4}. This endeavor entails working through nine tiers of data, each sourced differently, with the aim of producing and harmonizing agricultural SUA dataset for a diverse range of over 500 agricultural commodities. Compared to the FAO’s FBS, which report food and nutritional information for about 100 composite categories, in many cases combining primary and processed commodities (e.g., wheat and flour) within a single category (e.g., wheat and products), the constructed SUA explicitly trace the transformations between primary and processed commodities, while reporting nutritional details at the highly disaggregated level (over 500 commodities). In doing so, the constructed dataset substantially simplifies the FAOSTAT data processing steps by explicitly distinguishing food and nutritional supply at the individual commodity level and facilitating a straightforward mapping of the corresponding data to the global agroeconomic models. +Of particular significance is the `module_xfaostat_L105_DataConnectionToSUA`, which plays a pivotal role in harmonizing various FAOSTAT datasets to generate a cohesive set of agricultural supply and utilization accounts (SUA) data. This complex process is elucidated in \autoref{fig:Fig4}. This endeavor entails working through nine tiers of data, each sourced differently, with the aim of producing an harmonized agricultural SUA dataset for over 500 agricultural commodities. Compared to the FAO’s FBS, which report food and nutritional information for about 100 composite categories, in many cases combining primary and processed commodities (e.g., wheat and flour) within a single category (e.g., wheat and products), the constructed SUA explicitly trace the transformations between primary and processed commodities, while reporting nutritional details at a highly disaggregated level (over 500 commodities). In doing so, the constructed dataset substantially simplifies the FAOSTAT data processing steps by explicitly distinguishing food and nutritional supply at the individual commodity level and facilitating a straightforward mapping of the corresponding data to the global agroeconomic models. As an illustrative example, the first tier comprises 168 commodities, generated by combining production data from QCL, trade data from TM, and other essential balancing elements (such as opening and closing stocks, food and feed uses, and other industrial uses) from SCL. For a more comprehensive understanding of these procedures, we encourage an interested user to explore the mapping file, `FAO_items`. It is crucial to underscore the importance of these processing procedures, as raw FAOSTAT data often contains duplicated elements and inconsistencies among different datasets. For instance, trade data can be found in TCL, TM, SCL, and FBS, while production data exists in QCL and SCL (please see Table 1 for the corresponding dataset codes). @@ -163,11 +163,11 @@ As gcamfaostat is built upon the foundation of gcamdata and leverages the powerf In addition to generating data for modeling purposes, we also provide illustrative [examples](https://jgcri.github.io/gcamfaostat/articles/vignette_visualization.html) for visualizing the key data elements. Here are some examples for using the processed data to illustrate the connection of harvested area (\autoref{fig:Fig5}) and food calories (\autoref{fig:Fig6}) via supply utilization accounts (\autoref{fig:Fig7}). These figures focus on the 2013 – 2017 mean values as they are the base calibration years of GCAM. A user can change the years and mappings as desired. -![World area harvested (shares) grouped by GCAM crops based on the 2013 – 2017 mean values. The total harvested area is 1509 million hectares (Mha). \label{fig:Fig5}](./man/figures/Fig_WorldAreaHarvested.png){width=70%} +![World area harvested (shares) grouped by GCAM crops based on the 2013 – 2017 mean values. The total harvested area is 1509 million hectares (Mha). \label{fig:Fig5}](./man/figures/Fig_WorldAreaHarvested.png){width=100%} -![Food calories availability per capita per day grouped by GCAM regions and commodities based on the 2013 – 2017 mean values. The world average value is 2902 Kcal per capita per day (Kcal/ca/d). \label{fig:Fig6}](./man/figures/Fig_WorldFoodCaloriesRegPcPerDay.png){width=70%} +![Food calories availability per capita per day grouped by GCAM regions and commodities based on the 2013 – 2017 mean values. The world average value is 2902 Kcal per capita per day (Kcal/ca/d). \label{fig:Fig6}](./man/figures/Fig_WorldFoodCaloriesRegPcPerDay.png){width=100%} -![World supply utilization accounts for key commodities based on the 2013 – 2017 mean values. Note that negative values are used for demand categories and positive values are used for supply categories. Other uses include seed use, losses, and industrial use; residuals (mostly small) indicate the imbalance in the data. The total supply is equal to the total demand. \label{fig:Fig7}](./man/figures/Fig_WorldSUA.png){width=90%} +![World supply utilization accounts for key commodities based on the 2013 – 2017 mean values. Note that negative values are used for demand categories and positive values are used for supply categories. Other uses include seed use, losses, and industrial use; residuals (mostly small) indicate the imbalance in the data. The total supply is equal to the total demand. \label{fig:Fig7}](./man/figures/Fig_WorldSUA.png){width=100%} ## Other functions and capabilities @@ -213,14 +213,14 @@ Data development is never a once and for all task, and continued efforts are nee 1. **Sustain processing functions for updated raw data**: ensuring that our processing functions remain up-to-date when raw data undergoes revisions is imperative. 2. **Evaluate and enhance assumptions**: a critical examination of the assumptions utilized in processes like interpolation, extrapolation, aggregation, disaggregation, and mapping is essential and should be an ongoing endeavor. 3. **Revise assumptions in low-quality data zones**: regions and sectors with little or low-quality data require careful consideration. We will need to adjust our assumptions when improved data becomes available. -4. **Promoting broader applications**: leveraging data processed by gcamfaostat can significantly contribute to harmonizing input data in global agroeconomic modeling. Encouraging the utilization of this data and fostering collaboration to enhance data processing is of significant importance. +4. **Promoting broader applications**: leveraging data processed by gcamfaostat can significantly contribute to harmonizing input data in global agroeconomic modeling. Encouraging the utilization of this data and fostering collaboration to enhance data processing is crucial. 5. **Assess sensitivity in downstream applications**: understanding the sensitivity of downstream data applications, e.g., global agroeconomic projections, to upstream data processing assumptions is crucial. This awareness empowers us to make informed decisions and refinements. We welcome and value community contributions to gcamfaostat.Through collective and collaborative efforts, we hope to improve the interface between raw data, modeling community, and broader audience. We would be grateful for the feedback and suggestions on potential improvements of the developed data processing framework. # Acknowledgements -This research was supported by the US Department of Energy, Office of Science, as part of research in MultiSector Dynamics, Earth and Environmental System Modeling Program. The Pacific Northwest National Laboratory is operated for DOE by Battelle Memorial Institute under contract DE-AC05-76RL01830. Dr. Calvin is currently detailed to the National Aeronautics and Space Administration. Dr. Calvin’s contributions to this article occurred prior to her detail. The views expressed are her own and do not necessarily represent the views of the National Aeronautics and Space Administration or the United States Government. +This research was supported by the US Department of Energy, Office of Science, as part of research in MultiSector Dynamics, Earth and Environmental System Modeling Program. The Pacific Northwest National Laboratory is operated for DOE by Battelle Memorial Institute under contract DE-AC05-76RL01830. Dr. Calvin is currently detailed to the National Aeronautics and Space Administration. Dr. Calvin’s contributions to this article occurred prior to her detail. The views expressed are her own and do not necessarily represent the views of the National Aeronautics and Space Administration or the United States Government. We extend our sincere appreciation to Matthew Binsted and Page Kyle for their invaluable contributions. diff --git a/vignettes/vignette_getting_started.Rmd b/vignettes/vignette_getting_started.Rmd index 45bbdd59..4ee3269b 100644 --- a/vignettes/vignette_getting_started.Rmd +++ b/vignettes/vignette_getting_started.Rmd @@ -28,15 +28,15 @@ knitr::opts_chunk$set( ## Introduction -Global economic and multisector dynamic models have become pivotal tools for investigating complex interactions between human activities and the environment, as evident in recent research [@Doelman2022Quantifying;@Fujimori2022Land-based;@IPCC2022Annex;@Ven2023multimodel]. Agriculture and land use (AgLU) plays a critical role in these models, particularly when used to address key agroeconomic questions [@Graham2023Agricultural;@Yarlagadda2023Trade;@Zhang2023Agriculture;@Zhao2021Global;@Zhao2020critical]. Sound economic modeling hinges significantly upon the accessibility and quality of data [@Bruckner2019FABIO;@Calvin2022GMD;@Chepeliev2022JGEA]. The Food and Agriculture Organization Statistis (FAOSTAT) [@FAOSTAT2023FAOSTAT] serves as one of the key global data sources, offering open-access data on country-level agricultural production, land use, trade, food consumption, nutrient content, prices, and more. However, the raw data from FAOSTAT requires cleaning, balancing, and synthesis, involving assumptions such as interpolation and mapping, which can introduce uncertainties. In addition, some of the core datasets reported by FAOSTAT, such as FAO’s Food Balance Sheets (FBS), are compiled at a specific level of aggregation, combining together primary and processed commodities (e.g., wheat and flour), which creates additional data processing challenges for the agroeconomic modeling community [@Chepeliev2022JGEA]. It is noteworthy that each agroeconomic modeling team typically develops its own assumptions and methods to prepare and process FAOSTAT data [@bond2019gcamdata]. While largely overlooked, the uncertainty in the base data calibration approach likely contributed to the disparities in model outcomes [@Lampe2014AgMIP;@zhao2021role]. Hence, our motivation is to create an open-source tool (**`gcamfaostat`**) for the preparation, processing, and synthesis of FAOSTAT data for global agroeconomic modeling. The tool can also be valuable to a broader range of users interested in understanding global agriculture trends and dynamics, as it provides accessible and processed data and [visualization](https://jgcri.github.io/gcamfaostat/articles/vignette_visualization.html) functions. +Global economic and multisector dynamic models have become pivotal tools for investigating complex interactions between human activities and the environment, as evident in recent research [@Doelman2022Quantifying;@Fujimori2022Land-based;@IPCC2022Annex;@Ven2023multimodel]. Agriculture and land use (AgLU) plays a critical role in these models, particularly when used to address key agroeconomic questions [@Graham2023Agricultural;@Yarlagadda2023Trade;@Zhang2023Agriculture;@Zhao2021Global;@Zhao2020critical]. Sound economic modeling hinges significantly upon the accessibility and quality of data [@Bruckner2019FABIO;@Calvin2022GMD;@Chepeliev2022JGEA]. The Food and Agriculture Organization Statistis (FAOSTAT) [@FAOSTAT2023FAOSTAT] serves as one of the key global data sources, offering open-access data on country-level agricultural production, land use, trade, food consumption, nutrient content, prices, and more. However, the raw data from FAOSTAT requires cleaning, balancing, and synthesis, involving assumptions such as interpolation and mapping, which can introduce uncertainties. In addition, some of the core datasets reported by FAOSTAT, such as FAO’s Food Balance Sheets (FBS), are compiled at a specific level of aggregation, combining together primary and processed commodities (e.g., wheat and flour), which creates additional data processing challenges for the agroeconomic modeling community [@Chepeliev2022JGEA]. It is noteworthy that each agroeconomic modeling team typically develops its own assumptions and methods to prepare and process FAOSTAT data [@bond2019gcamdata]. While largely overlooked, the uncertainty in the base data calibration approach likely contribute to the disparities in model outcomes [@Lampe2014AgMIP;@zhao2021role]. Hence, our motivation is to create an open-source tool (**`gcamfaostat`**) for the preparation, processing, and synthesis of FAOSTAT data for global agroeconomic modeling. The tool can also be valuable to a broader range of users interested in understanding global agriculture trends and dynamics, as it provides accessible and processed data and [visualization](https://jgcri.github.io/gcamfaostat/articles/vignette_visualization.html) functions. *** ## The gaps **`gcamfaostat`** fills -[**Figure 1**](#Fig1) shows a standard framework of using FAOSTAT data in GCAM. GCAM is a widely recognized global economic and multisector dynamic model complemented by the gcamdata R package, which serves as its data processing system. Particularly, gcamdata includes modules (data processing chunks) and functions to convert raw data inputs into hundreds of XML files needed by GCAM [@bond2019gcamdata]. As an illustration, in the latest GCAM version, GCAM v7 [@bond_lamberty_2023], about 280 XML files, with a combined size of 4.1 GB, are generated. Although AgLU-related XMLs represent only about 10% of the total number of files, they contribute over 50% in size (~2.1 GB). The majority of AgLU-related data, whether directly or indirectly, rely on raw data sourced from the FAOSTAT. +[**Figure 1**](#Fig1) shows a standard framework of using FAOSTAT data in GCAM. GCAM is a widely recognized global economic and multisector dynamic model complemented by the gcamdata R package, which serves as its data processing system. Particularly, gcamdata includes modules (data processing chunks) and functions to convert raw data inputs into hundreds of XML input files used by GCAM [@bond2019gcamdata]. As an illustration, in the latest GCAM version, GCAM v7 [@bond_lamberty_2023], about 280 XML files, with a combined size of 4.1 GB, are generated. Although AgLU-related XMLs represent only about 10% of the total number of files, they contribute over 50% in size (~2.1 GB). The majority of AgLU-related data, whether directly or indirectly, rely on raw data sourced from the FAOSTAT. -Nonetheless, the FAOSTAT data employed within gcamdata has traditionally involved manual downloads and may have undergone preprocessing. In light of the increasing data needs, maintaining the FAOSTAT data processing tasks in gcamdata has become increasingly challenging. In addition, the processing of FAOSTAT data in the AgLU modules of gcamdata is tailored specifically for GCAM. Consequently, the integration of FAOSTAT data updates has proven to be a non-trivial task, and the data processed by the AgLU module has limited applicability in other modeling contexts [@zhao_cmp360]. The **`gcamfaostat`** package is aimed to address these limitations ([**Figure 2**](#Fig2)). The targeted approach incorporates data preparation, processing, and synthesizing capabilities within a dedicated package, gcamfaostat, while regional and sectoral aggregation functions in the model data system are implemented using standalone routines within the gcamdata package. This strategy not only ensures the streamlined operation of **`gcamfaostat`** but also contributes to keeping model data system lightweight and more straightforward to maintain. +Nonetheless, the FAOSTAT data employed within gcamdata has traditionally involved manual downloads and may have undergone preprocessing. In light of the increasing data needs, maintaining the FAOSTAT data processing tasks in gcamdata has become increasingly challenging. In addition, the processing of FAOSTAT data in the AgLU modules of gcamdata is tailored specifically for GCAM. Consequently, the integration of FAOSTAT data updates has proven to be a non-trivial task, and the data processed by the AgLU module has limited applicability in other modeling contexts [@zhao_cmp360]. The **`gcamfaostat`** package aims to address these limitations ([**Figure 2**](#Fig2)). The targeted approach incorporates data preparation, processing, and synthesis capabilities within a dedicated package, gcamfaostat, while regional and sectoral aggregation functions in the model data system are implemented using standalone routines within the gcamdata package. This strategy not only ensures the streamlined operation of **`gcamfaostat`** but also contributes to keeping model data system lightweight and more straightforward to maintain.
diff --git a/vignettes/vignette_news.Rmd b/vignettes/vignette_news.Rmd index 5df2e4d1..a58b8510 100644 --- a/vignettes/vignette_news.Rmd +++ b/vignettes/vignette_news.Rmd @@ -15,13 +15,20 @@ knitr::opts_chunk$set( ) ``` -# gcamfaostat 1.0.0 +# November 2023 updates -**Beta release** -* The beta release of gcamfaostat 1.0.0 includes the data generated for the Global Change Analysis Model v7.0 [GCAM v7.0](https://github.com/JGCRI/gcam-core/releases/tag/gcam-v7.0). -* +# **JOSS submission** +2023-11-10 + +* gcamfaostat is now submitted to JOSS for review and publication +* gcamfaostat v1.0.0 includes the data generated for the Global Change Analysis Model v7.0 [GCAM v7.0](https://github.com/JGCRI/gcam-core/releases/tag/gcam-v7.0). +* We will make an official release after the JOSS review. + +2023-11-09 +# FAOSTAT (in the recent Oct release) provide 2021 data +# We will now update gcamfaostat to include the latest data and for GCAM BYU +# A new **BYU branch** is created for this task -# November 2023 updates 2023-11-08 * All Github Action test passed. Badges added! @@ -67,7 +74,7 @@ ToDo before JOSS submission * Compare the results from the new primary commodity equivalent method with the FBS data compiled using FAOSTAT method (never disclosed). -**Raw data archive** +# **Raw data archive** 2023-08-19 The source data downloaded from FAOSTAT is archived at a [Zenodo repository]( https://zenodo.org/deposit/8260225). diff --git a/vignettes/vignette_processing_flow.Rmd b/vignettes/vignette_processing_flow.Rmd index feac5c86..aa6edd7c 100644 --- a/vignettes/vignette_processing_flow.Rmd +++ b/vignettes/vignette_processing_flow.Rmd @@ -31,7 +31,7 @@ The architecture of gcamfaostat processing modules is depicted in the [**Figure ### Data synthesizing in a key module -Of particular significance is the `module_xfaostat_L105_DataConnectionToSUA`, which plays a pivotal role in harmonizing various FAOSTAT datasets to generate a cohesive set of agricultural supply and utilization accounts (SUA) data. This complex process is elucidated in the following [**Figure 2**](#Fig2). This endeavor entails working through nine tiers of data, each sourced differently, with the aim of producing and harmonizing agricultural SUA dataset for a diverse range of over 500 agricultural commodities. Compared to the FAO’s FBS, which report food and nutritional information for about 100 composite categories, in many cases combining primary and processed commodities (e.g., wheat and flour) within a single category (e.g., wheat and products), the constructed SUA explicitly trace the transformations between primary and processed commodities, while reporting nutritional details at the highly disaggregated level (over 500 commodities). In doing so, the constructed dataset substantially simplifies the FAOSTAT data processing steps by explicitly distinguishing food and nutritional supply at the individual commodity level and facilitating a straightforward mapping of the corresponding data to the global agroeconomic models. +Of particular significance is the `module_xfaostat_L105_DataConnectionToSUA`, which plays a pivotal role in harmonizing various FAOSTAT datasets to generate a cohesive set of agricultural supply and utilization accounts (SUA) data. This complex process is elucidated in the following [**Figure 2**](#Fig2). This endeavor entails working through nine tiers of data, each sourced differently, with the aim of producing an harmonized agricultural SUA dataset for over 500 agricultural commodities. Compared to the FAO’s FBS, which report food and nutritional information for about 100 composite categories, in many cases combining primary and processed commodities (e.g., wheat and flour) within a single category (e.g., wheat and products), the constructed SUA explicitly trace the transformations between primary and processed commodities, while reporting nutritional details at a highly disaggregated level (over 500 commodities). In doing so, the constructed dataset substantially simplifies the FAOSTAT data processing steps by explicitly distinguishing food and nutritional supply at the individual commodity level and facilitating a straightforward mapping of the corresponding data to the global agroeconomic models. As an illustrative example, the first tier comprises 168 commodities, generated by combining production data from QCL, trade data from TM, and other essential balancing elements (such as opening and closing stocks, food and feed uses, and other industrial uses) from SCL. For a more comprehensive understanding of these procedures, we encourage an interested user to explore the mapping file, FAO_items. It is crucial to underscore the importance of these processing procedures, as raw FAOSTAT data often contains duplicated elements and inconsistencies among different datasets. For instance, trade data can be found in TCL, TM, SCL, and FBS, while production data exists in QCL and SCL.
diff --git a/vignettes/vignette_use_cases.Rmd b/vignettes/vignette_use_cases.Rmd index f5563af2..da6fa3f8 100644 --- a/vignettes/vignette_use_cases.Rmd +++ b/vignettes/vignette_use_cases.Rmd @@ -55,7 +55,7 @@ Since the 1970s, the number of countries in the world has increased due to the d [`FAOSTAT_AREA_RM_NONEXIST`](https://jgcri.github.io/gcamfaostat/reference/FAOSTAT_AREA_RM_NONEXIST.html) -* The function removes nonexistent FAO region using area_code, e.g., USSR after 1991. +* The function removes nonexistent FAO regions using area_code, e.g., USSR after 1991. * All nonexistent countries due to dissolution are removed by default. * Small regions/areas with low data quality can also be removed using this function.

Dataset Code Dataset Name Exist_Local