README.md

Glue Connector

This connector extracts technical metadata from AWS Glue using the boto3 library.

Setup

We recommend creating a dedicated AWS IAM user for the crawler with limited permissions based on the following IAM policy:

{
    "Version": "2012-10-17",
    "Statement":
    [
        {
            "Effect": "Allow",
            "Action":
            [
                "glue:GetTable",
                "glue:GetTables",
                "glue:GetDatabase",
                "glue:GetDataBases"
            ],
            "Resource":
            [
                "*"
            ]
        }
    ]
}

Config File

Create a YAML config file based on the following template.

Required Configurations

You must specify an AWS user credential to access Glue API. You can also specify a role ARN and let the connector assume the role before accessing AWS APIs.

aws:
  access_key_id: <aws_access_key_id>
  secret_access_key: <aws_secret_access_key>
  region_name: <aws_region_name>
  assume_role_arn: <aws_role_arn>  # If using IAM role

Optional Configurations

Output Destination

See Output Config for more information.

Testing

Follow the Installation instructions to install metaphor-connectors in your environment (or virtualenv).

Run the following command to test the connector locally:

metaphor glue <config_file>

Manually verify the output after the run finishes.