Adding an Aliyun DataWorks MaxCompute Connection

Prerequisites

• A user with sufficient permissions is required to establish a connection with Aliyun DataWorks MaxCompute.
• Zeenea traffic flows towards the data source must be open.

note

A configuration template can be downloaded here: aliyun-dataworks-maxcompute.conf

Installing the Plugin

The Aliyun DataWorks MaxCompute plugin can be downloaded here: Zeenea Connector Downloads.

For more information on how to install a plugin, please refer to the following article: Installing and Configuring Connectors as a Plugin.

Declaring the Connection

Creating and configuring connectors is done through a dedicated configuration file located in the /connections folder of the relevant scanner.

Read more: Managing Connections

In order to establish a connection with a Aliyun DataWorks MaxCompute instance, specifying the following parameters in the dedicated file is required:

Parameter	Expected value
name	The name that will be displayed to catalog users for this connection.
code	The unique identifier of the connection on the Zeenea platform. Once registered on the platform, this code must not be modified or the connection will be considered as new and the old one removed from the scanner.
connector_id	The connector type to use for the connection. Here, the value must be aliyun-dataworks-maxcompute and this value must not be modified.
connection.region_id	The region ID. The complete list is available here.
connection.access_key_id	The user's access key id. This parameter is optional, the key can be provided through the environment variable ALIBABA_CLOUD_ACCESS_KEY_ID. If both are defined, the environment variable will be used.
connection.access_key_secret	The secret associated to the access key. This parameter is optional, the key can be provided through the environment variable ALIBABA_CLOUD_ACCESS_KEY_SECRET. If both are defined, the environment variable will be used. Example: export ALIBABA_CLOUD_ACCESS_KEY_SECRET="abc654dqzER54C15674EZcazd"
Quota
quota.max_retry	Maximum number of retries when a request encounters a quota expiration error. Default value: 3.
quota.timeout_minute	Maximum waiting time when waiting for the availability of a quota, in minutes. Default value: 1 minute.
quota.call_per_min	Calls per minute quota value. Default value: 50.
Proxy
proxy.scheme	Depending on the proxy, http or https
proxy.hostname	Proxy address
proxy.port	Proxy port
proxy.username	Proxy username
proxy.password	Proxy account password

User Permissions

To collect the items, the technical user must be added to each project that needs cataloging.

Data Extraction

To extract information, the connector runs requests on following services:

• ListProjects: Retrieve all projects.
• GetMetaDBTableList: Retrieve all project's tables.
• GetMetaTableBasicInfo: Retrieve table's info.
• GetMetaTableColumn: Retrieve all table's columns.
• GetMetaTableLineage: Retrieve all table's lineage.

Collected Metadata

Inventory

Will collect the list of tables and views accessible by the user.

Dataset

A dataset can be a table or a view.

• Name
• Source Description
• Technical Data:
  • Type
  • Creation Date
  • Last Modification Date

Field

Dataset field.

• Name
• Source Description
• Type
• Can be null: Not supported. Default value FALSE.
• Multivalued: Not supported. Default value FALSE.
• Primary Key: Depending on the field's "Primary Key" attribute.
• Technical Data:
  • Native type

Data Processes

To represent the data flow between tables, a Zeenea Data Process will be created for a table and its ancestors.

• Name: import dataset_name

Unique Identification Keys

A key is associated with each item of the catalog. When the object comes from an external system, the key is built and provided by the connector.

More information about how it works can be found here: Identification Keys.

Object	Identifier Key	Description
Dataset	code/table ID	code: Unique identifier of the connection noted in the configuration file table ID: Unique ID defined by the source system
Field	code/column ID	code: Unique identifier of the connection noted in the configuration file column ID: Unique ID defined by the source system
Data process	code/import/output table ID	code: Unique identifier of the connection noted in the configuration file output table ID: Unique ID of the output table