Tetra Agent Integration and Connectors (For TDP Versions 3.0 and Below)

📘

NOTE

This topic is for TDP Version 3.0 and below.

After the Tetra Agent extracts copies of the data files from the data source, it sends those files to the Tetra Data Platform. This topic provides details on how to configure integrations and connectors to transfer the data.

Tetra Agent Integration and Connector Overview

TetraScience provides several different ways for Tetra Agents to integrate with the Tetra Data Platform.

Ways that Tetra Agents Can Integrate with TDP

Agent machine can only connect to the Tetra Data HubAgent machine can connect directly to the Cloud
Agent uses AWS APIs (S3, CloudWatch, etc)S3 via proxy

1. Agent connects to GDC, sends it the agent ID and JSON Web Token.
2. GDC responds with temporary credentials and the proxy location.
3. Agent uses temporary credentials and proxy to upload to S3.
S3 without proxy
(S3 Direct Upload option)

1. Agent connects to GDC or UDI, sends it agent ID and JSON Web Token.
2. GDC or UDI responds with temporary credentials.
3. Agent uses temporary credentials to upload to S3.
Agent uses TetraScience's simplified wrapperStandard

1. Agent connects to GDC, sends it Agent ID and content.
UDI

1. Agent connects to UDI, sends it Agent ID and JSON Web Token, and content.

Selecting a Connector: GDC or UDI

To determine which type of connector to use, you’ll need to know two things:

  • Is the Tetra Agent installed on a machine that has internet access?
  • Are the files to be transferred to the data lake large (more than 500 MB) or small (less than 500 MB)?

Connector Decision Table

Agent Internet AccessSmall Files (less than 500 mb)Large Files (more than 500 mb)
Agent HAS Internet AccessUDI (API upload)UDI (S3 upload)
Agent DOES NOT HAVE Internet AccessGDC/DataHub (API Upload)GDC/DataHub (S3 Upload) with the L7 Proxy Connector
  • If the agent has internet access, and files are small, use the User Defined Integration (UDI) connector.
  • If the agent has internet access, and the files are large, use UDI, and make sure that you have the S3 upload configuration enabled.
  • If the agent does not have internet access, and the files are small, use the Generic Data Connector (GDC), which should be on the Data Hub.
  • If the agent does not have internet access, and the files are large, use the GDC, which should be on the Data Hub. You will also need to use the L7 Proxy Connector. The L7 Proxy Connector (L7C) is a simple proxy server that uploads files directly to the data lake. L7C accommodates large files. The L7 Proxy Connector should also be set up on the Data Hub.
  • The Agent's host machine will need to be on Windows 10 or newer, or Windows Server 2016 for the Cloud Data Connector to connect to it properly.

Connector URL

After determining which connector to use you will need to specify the Connection URL. Examples of the URL from GDC or UDI as below

Connector/Integration Configuration Instructions

Generic Data Connector (GDC)

Tetra Agents can be integrated with the Tetra Data Platform through Tetra Data Hub. The Tetra Data Hub is an on-premise server that facilitates securely transferring data extracted from a Tetra Agent to the Tetra Data Platform. The Agent connects to the Data Hub through a data connector.

Prerequisites

Procedures

Because the Tetra Agent transfers data with a Tetra Data Hub Connector, you must register the Tetra Agent first on the Tetra Data Platform with a specific connector.

Here's how to add a Tetra Agent to your existing connector on the Tetra Data Platform:

  1. Use your web browser to log into the Tetra Data Platform.
  2. Click the menu in the upper left corner to view the menu options. Select Data Sources, then Data Hub.
19051905

Data Hub Menu Option

  1. In the Data Management page, find the connector you want to add the Tetra Agent to. Click on the "menu" icon (1), then click "Edit" (2).
19051905

Edit the Connector

  1. Scroll to the Agent section, then click Add Agent.
19051905

Add a Tetra Agent to the Connector

  1. The Agent ID is auto-generated by the Tetra Data Platform. You just need to give the Agent a name. Optionally, you can also specify Agent metadata and tags for quick identification in the Data Management page.

  2. Click Save.

19051905
  1. The new Agent appears with the connector on the Data Management page. The status of the Agent changes to online when the Agent service starts and successfully connects to the connector.
19051905

New Agent Appears with the Connector

User-Defined Integration (UDI)

Besides the connection through the Data Hub/Data Connector, the Agent can directly upload the files to the Tetra Data Platform by using the so-called User-defined Integration API.

To set up User-defined Integration API to be used by an Agent, the user needs to get a valid Agent Id and an API Key. The following procedures illustrate how to get Agent Id and API Key from the TetraScience Integration Platform.

Pre-requisites

  • Client Organization Slug (orgSlug) is created on Tetra Data Platform
  • A user account is created in that Organization Slug

Procedures

  1. Log in to Tetra Data Platform using the user account
423423
  1. Navigate to User Defined Integration from the menu, and then click Add User Defined Integration on the right corner
14451445
  1. Provide Integration Name and click Create
637637
  1. Provide detailed information.

Agent Id is generated as default.

4.1 Select Source Type

Source Type is mandatory. The default is unknown

792792

4.2 Provide Name and Description

791791

4.3 Provide Meta Data and Tags

Provide metadata and tags if it is needed and click Finish to complete the

790790

4.4 Agent is created.

The Agent Id will be used in the Agent configuration

12751275
  1. Retrieve API Key

The API Key is associated with every user account. The user can see her API Key by following the procedures as below:

  • Click the user name icon on the right corner.
  • Click Account to display Account Details
  • API Key will be shown on the detail page.
15641564

Now the user has both Agent Id and API Key. It is ready to set up the Agent to upload the files to the Tetra Data Platform.

  1. Retrieve JWT Token

Besides using TetraScience API Key, the user can attach a JWT token to the header when setting up a TetraScience Agent using UDI.

JWT token is associated with Service User. The user can get a valid JWT token by either creating a new Service User or generating the token from an existing Service User.

The user can follow the procedures below to retrieve the token.

6.1 Navigate to http://url_of_tetrascience_data_platform/account/organization

6.2 Click Add Service Account or select an existing Service Account

793793

6.3 Click the menu on the right of the Service User record, and select the Generate token option.

858858

6.4 A JWT token is created and displayed in a model window. The user can copy the token and save it for setting up a TetraScience Agent to connect TetraScience Data Platform

582582

Please note, the user can specify the expiring day when generating the token. The maximum is 180 days.

L7 Proxy Connector

It is optional to create an L7 Proxy Connector along with Generic Data Connector. The purpose of L7 Proxy Connector to allow end-user to upload large files (size is larger than 500 MB) to TetraScience Data Platform using the Generic Data Connector option

❗️

For large files (more than 500MB)

Remember to select S3 direct upload in the agent and point to the L7 Proxy connector. L7 Proxy without S3 direct upload will fail at uploading large files.

The end-user can follow the instruction below to create an L7 Proxy Connector.

  1. Go to TetraScience platform page on your browser
  2. Click on the upper left corner to bring up the sidebar and click "Data Hub"
  3. Click Add Connector
  4. Select L7 Proxy Connector from the dropdown
10131013
  1. Fill in the Name, and provide the Port number. The default is 3128. Please double check with System Administrator to make sure this port is available
10221022
  1. Click Save, then wait a couple of minutes. The L7 Proxy Connector will show online. Now, it is ready.
12231223