Develop script to update metadata locally

[theamarks]

Descripton

Current workflow for submitting data and metatdata to KNB is a hacky hybrid approach. We created the data package via website GUI and then pull down the package locally to change public viewing options. Moving forward we would like to provided more detailed and compressive information via scripts about the 2 KNB dataset we now maintain (doi:10.5063/F1CZ35N7 & doi:10.5063/F1862DXT). This would include:

• build EML locally
• link existing knb_submit.R script to use locally build EML file to update datapackage
• document object provenance
• Provide unzipped files for further transparency and building provenance - .parquet probably
• Transfer data in a more efficient way (not sure on the categories laid out here)
  • aws-sync
  • SFTP

Considerations

• Where do we want this EML build script to live? Do we include metadata generation in the artis-model pipeline? Would make sense if we are working towards automating more and more of the pipeline.
• Can we feed the EML build script model configuration values from 00-* scripts? These might eventually be in a configuration YAML, can we use that in conjunction with the EML script to avoid duplicating definitions? Possibly like this:

# Load configuration from YAML file
config <- yaml::read_yaml("config.yaml")

# Set project directory and metadata file destination
project_dir <- config$paths$project_dir
output_dir <- file.path(project_dir, "metadata")
dir.create(output_dir, showWarnings = FALSE, recursive = TRUE)

# Extract metadata information from config
artis_version <- config$artis_version
prod_data_type <- config$prod_data_type
estimate_data_type <- config$estimate_data_type

# Define basic metadata information
title <- glue::glue("ARTIS {artis_version}: Aquatic Resource Trade in Species Model Data")

# Create a person object for the creator - you could have these in the config too
creator <- eml_person(
  givenName = config$metadata$creator$given_name,
  surName = config$metadata$creator$surname,
  email = config$metadata$creator$email,
  organizationName = config$metadata$creator$organization,
  userId = list(directory = "https://orcid.org", userId = config$metadata$creator$orcid)
)

Resources

• ADC scalable computing coursebook - Publishing Data Section

[theamarks]

Positron Assistant Draft template for building ARTIS EML and example config.yaml file:

# Load required packages
library(EML)
library(tidyverse)
library(uuid)

# Set project directory and metadata file destination
project_dir <- "path/to/your/project"
output_dir <- file.path(project_dir, "metadata")
dir.create(output_dir, showWarnings = FALSE, recursive = TRUE)

# Define basic metadata information
title <- "ARTIS v1.2.0: Aquatic Resource Trade in Species Model Data"

# Create a person object for the creator
creator <- eml_person(
  givenName = "Your Name",
  surName = "Your Surname",
  email = "your.email@example.com",
  organizationName = "Your Organization",
  userId = list(directory = "https://orcid.org", userId = "Your-ORCID-ID")
)

# Add additional contacts/contributors if needed
additional_creator <- eml_person(
  givenName = "Colleague",
  surName = "Name",
  email = "colleague.email@example.com",
  organizationName = "Their Organization",
  userId = list(directory = "https://orcid.org", userId = "Their-ORCID-ID")
)

# Define abstract
abstract <- set_TextType(
  "This dataset contains outputs from the ARTIS v1.2.0 (Aquatic Resource Trade in Species) 
  model which tracks global seafood trade flows. The data includes both trade and consumption 
  tables derived from production, trade, and conversion factor data. The model includes data 
  for marine and inland capture and aquaculture production, with trade data sourced from the 
  BACI database and supplemented with Sea Around Us (SAU) data."
)

# Define intellectual rights/license
intellectualRights <- set_TextType(
  "This dataset is released under a Creative Commons Attribution 4.0 International License."
)

# Define keywords
keywords <- set_keywordSet(
  keywordThesaurus = "ARTIS controlled vocabulary",
  keywords = c("seafood trade", "fisheries", "aquaculture", "consumption", "global trade flow")
)

# Define geographic coverage
geographicCoverage <- set_coverage(
  geographicDescription = "Global coverage of seafood trade flows",
  west = -180,
  east = 180,
  north = 90,
  south = -90
)

# Define temporal coverage
temporalCoverage <- set_coverage(
  beginDate = "1996-01-01", 
  endDate = "2020-12-31"
)

# Combine coverage elements
coverage <- list(
  geographicCoverage = geographicCoverage,
  temporalCoverage = temporalCoverage
)

# Define taxonomic coverage
taxonomicCoverage <- set_taxonomicCoverage(
  taxonRankName = c("Kingdom", "Phylum", "Class"),
  taxonRankValue = c("Animalia", "Chordata", "Actinopterygii")
)

# Add taxonomic coverage to the coverage list
coverage$taxonomicCoverage <- taxonomicCoverage

# Define methods - data collection
methods <- set_methods(
  methodStep = list(
    list(
      description = set_TextType(
        "The ARTIS model v1.2.0 combines data from the Food and Agriculture Organization (FAO) 
        and Sea Around Us (SAU) databases for production data. Trade data is sourced from BACI trade database.
        The model applies conversion factors to standardize all quantities to live-weight equivalents."
      ),
      instrumentation = "Software processing using R version 4.4.2 with custom scripts"
    )
  )
)

# Define attributes for the trade data table
trade_attributes <- data.frame(
  attributeName = c("year", "hs_version", "exporter", "importer", "sciname", 
                    "habitat", "prod_method", "quantity", "trade_value"),
  attributeDefinition = c(
    "Year of trade flow",
    "Harmonized System version (HS96, HS02, etc)",
    "ISO3 code of exporting country",
    "ISO3 code of importing country", 
    "Scientific name of species or taxonomic group",
    "Habitat classification (marine, inland, diadromous)",
    "Production method (capture, aquaculture)",
    "Trade quantity in live-weight equivalent (tonnes)",
    "Trade value in USD"
  ),
  formatString = c("YYYY", "", "ISO3", "ISO3", "", "", "", "", ""),
  unit = c("year", NA, NA, NA, NA, NA, NA, "tonnes", "USD"),
  numberType = c("integer", NA, NA, NA, NA, NA, NA, "real", "real"),
  definition = c(NA, NA, NA, NA, NA, NA, NA, "Live-weight equivalent", "Current USD"),
  stringsAsFactors = FALSE
)

# Define attributes for the consumption data table
consumption_attributes <- data.frame(
  attributeName = c("year", "country", "sciname", "habitat", 
                    "prod_method", "source", "quantity", "per_capita"),
  attributeDefinition = c(
    "Year of consumption",
    "ISO3 code of consuming country", 
    "Scientific name of species or taxonomic group",
    "Habitat classification (marine, inland, diadromous)",
    "Production method (capture, aquaculture)",
    "Source of consumption (domestic or imported)",
    "Consumption quantity in live-weight equivalent (tonnes)",
    "Per capita consumption (kg per person per year)"
  ),
  formatString = c("YYYY", "ISO3", "", "", "", "", "", ""),
  unit = c("year", NA, NA, NA, NA, NA, "tonnes", "kg/person/year"),
  numberType = c("integer", NA, NA, NA, NA, NA, "real", "real"),
  definition = c(NA, NA, NA, NA, NA, NA, "Live-weight equivalent", "Annual per capita"),
  stringsAsFactors = FALSE
)

# Set attribute lists
trade_attributeList <- set_attributes(trade_attributes, 
                                      col_classes = c("numeric", "character", "character", "character", 
                                                     "character", "character", "character", "numeric", "numeric"))

consumption_attributeList <- set_attributes(consumption_attributes, 
                                           col_classes = c("numeric", "character", "character", "character", 
                                                          "character", "character", "numeric", "numeric"))

# Define physical metadata for the trade file
trade_physical <- set_physical(
  objectName = "ARTIS_v1.2.0_trade_SAU_mid_all_HS_yrs_2025-11-19.parquet",
  size = list(size = "estimate_file_size_here", unit = "byte"),
  dataFormat = set_dataFormat(
    externallyDefinedFormat = list(
      formatName = "Apache Parquet"
    )
  ),
  authentication = list(
    authentication = "MD5 checksum_here_if_available"
  )
)

# Define physical metadata for the consumption file
consumption_physical <- set_physical(
  objectName = "ARTIS_v1.2.0_consumption_SAU_mid_2025-11-19.parquet",
  size = list(size = "estimate_file_size_here", unit = "byte"),
  dataFormat = set_dataFormat(
    externallyDefinedFormat = list(
      formatName = "Apache Parquet"
    )
  ),
  authentication = list(
    authentication = "MD5 checksum_here_if_available"
  )
)

# Create the dataTable objects
trade_dataTable <- new("dataTable",
                       entityName = "ARTIS_v1.2.0_trade_SAU_mid_all_HS_yrs_2025-11-19.parquet",
                       entityDescription = "Trade flow data from ARTIS v1.2.0",
                       physical = trade_physical,
                       attributeList = trade_attributeList)

consumption_dataTable <- new("dataTable",
                            entityName = "ARTIS_v1.2.0_consumption_SAU_mid_2025-11-19.parquet",
                            entityDescription = "Consumption data from ARTIS v1.2.0",
                            physical = consumption_physical,
                            attributeList = consumption_attributeList)

# Create the dataset object
dataset <- new("dataset",
               title = title,
               creator = list(creator),
               metadataProvider = list(creator),
               associatedParty = list(additional_creator),
               pubDate = as.character(Sys.Date()),
               abstract = abstract,
               intellectualRights = intellectualRights,
               keywordSet = list(keywords),
               coverage = coverage,
               methods = methods,
               dataTable = list(trade_dataTable, consumption_dataTable))

# Generate a unique ID for the EML package
packageId <- paste0("artis.1.2.0.", format(Sys.time(), "%Y%m%d%H%M%S"))

# Create the EML object
eml <- new("eml",
           packageId = packageId,
           system = "uuid",
           dataset = dataset)

# Validate the EML document
eml_validate(eml)

# Write the EML document
write_eml(eml, file.path(output_dir, "artis_metadata.xml"))

cat("EML metadata document has been created at:", file.path(output_dir, "artis_metadata.xml"))

Example config.yaml

# ARTIS model configuration
artis_version: "v1.2.0"
start_date: "2025-11-20"
baci_version: "202201"
prod_data_type: "SAU"  # "SAU" or "FAO"
estimate_data_type: "midpoint"  # "min", "midpoint", or "max"
years: [1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020]
hs_versions:
  - "96"
  - "02"
  - "07" 
  - "12"
  - "17"

# Paths configuration
paths:
  project_dir: "~/Documents/UW-SAFS/ARTIS"
  datadir_raw: "~/Documents/UW-SAFS/ARTIS/data/model_inputs_raw_1.2.0_FAO"
  datadir: "~/Documents/UW-SAFS/ARTIS/data/model_inputs_1.2.0_FAO"
  tradedatadir: "baci_raw/baci_202201"
  outdir: "~/Documents/UW-SAFS/ARTIS/data/outputs_1.2.0_FAO"
  outputs_combined: "~/Documents/UW-SAFS/ARTIS/data/outputs_1.2.0_SAU_2025-11-14/outputs_combined"

# Metadata information
metadata:
  creator:
    given_name: "Thea"
    surname: "Marks"
    email: "theamarks@example.com"
    organization: "University of Washington"
    orcid: "0000-0000-0000-0000"
  contributors:
    - given_name: "Jessica"
      surname: "Gephart"
      email: "jessica.gephart@example.com"
      organization: "American University"
      orcid: "0000-0003-1957-1432"
  license: "This dataset is released under a Creative Commons Attribution 4.0 International License."
  keywords:
    - "seafood trade"
    - "fisheries"
    - "aquaculture"
    - "consumption"
    - "global trade flow"
    - "ARTIS"
  methods:
    description: "The ARTIS model combines data from the Food and Agriculture Organization (FAO) and Sea Around Us (SAU) databases for production data. Trade data is sourced from BACI trade database. The model applies conversion factors to standardize all quantities to live-weight equivalents."