Adding an AWS Glue (ETL) Connection

Prerequisites

• A user with sufficient permissions is required to establish a connection with AWS Glue.
• Zeenea traffic flows towards the database must be open.

The Agent's host server must have sufficient credentials to connect to AWS Glue; in this case, available authentication methods are:

• Instance Role
• Environment Variable
• Configuration File

Target	Protocol	Usual Ports
AWS Glue	HTTP	443

note

A configuration template can be downloaded here: aws-glue-etl.conf

Supported Versions

The AWS Glue connector was successfully tested with the online application.

Installing the Plugin

The AWS Glue plugin can be downloaded here: Zeenea Connector Downloads.

For more information on how to install a plugin, please refer to the following article: Installing and Configuring Connectors as a Plugin.

Declaring the Connection

Creating and configuring connectors is done through a dedicated configuration file located in the /connections folder of the relevant scanner.

Read more: Managing Connections

In order to establish a connection with an AWS Glue instance, specifying the following parameters in the dedicated file is required:

Parameter	Expected value
name	The name that will be displayed to catalog users for this connection.
code	The unique identifier of the connection on the Zeenea platform. Once registered on the platform, this code must not be modified or the connection will be considered as new and the old one removed from the scanner.
connector_id	The type of connector to be used for the connection. Here, the value must be aws-glue-etl and this value must not be modified.
connection.aws.url	(Optional) Fill this setting if you wish to use an Glue instance other than Amazon's.
connection.aws.access_key_id	AWS Glue Access Key Identifier
connection.aws.secret_access_key	AWS Glue Secret Access Key
connection.aws.region	AWS region
connection.aws.profile	AWS Profile for authentication
connection.fetch_page_size	(Advanced) define the size of batch of items loaded by each request in inventory. Since version 1.0.3
tls.truststore.path	The Trust Store file path. This file must be provided in case TLS encryption is activated (protocol https) and when certificates of AWS servers are delivered by a specific authority. It must contain the certification chain.
tls.truststore.password	Password of the trust store file
tls.truststore.type	Type of the trust store file. (PKCS12 or JKS`). Default value is discovered from the file extension.
proxy.scheme	Depending on the proxy, http or https
proxy.hostname	Proxy address
proxy.port	Proxy port
proxy.username	Proxy username
proxy.password	Proxy account password

User Permissions

In order to collect metadata, the running user's permissions must allow them to access and read jobs that need cataloging.

Roles

The user must be able to run the following actions on the target bucket and the objects it contains (JSON format):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "glue:GetDataflowGraph",
        "glue:GetJob",
        "glue:GetTables",
        "glue:GetDataTables"
      ],
      "Resource": "*"
    }
  ]
}

Data Extraction

To extract information, the connector runs requests on AWS API to collect jobs and metadata.

IMPORTANT

For lineage information to be exposed by the AWS API, it is necessary for Glue jobs to be auto-generated by Glue itself from the Jobs (Legacy) section. Indeed, the metadata transmitted by the API are based on the automatic and standardized comments written in the scripts which are only present with this method.

Collected Metadata

Synchronization

The connector will synchronize all transformations identified in the project and automatically represent them in the catalog.

Lineage

The AWS Glue connector is able to retrieve the lineage between datasets that have been imported to the catalog. Datasets from other connections must have been previously imported to the catalog to be linked to data processes. This feature is available for datasets imported from the AWS Glue (Datacatalog) connector and an additional parameter is needed in the affected AWS Glue (Datacatalog) connection configuration, as detailed below:

alias = ["region/database"]

In this instance, variables region and database need to be replaced with the actual AWS Glue (Datacatalog) values.

Data Processes

A data process is the representation of a Glue job.

• Name
• Source Description
• Technical Data:
  • Creation Date
  • Modification Date
  • Worker Type
  • Glue Version
  • Script location

Unique Identification Keys

A key is associated with each item of the catalog. When the object comes from an external system, the key is built and provided by the connector.

Read more: Identification Keys

Object	Identifier Key	Description
Data process	code/region/job name	code: Unique identifier of the connection noted in the configuration file region: AWS Job's region job name