This topic is part of the Empower Data Science Link (EDSL) Quick Start Guide. If you need more details about the topics discussed in this section, see the For More Information section at the bottom of this page.
In this step, you’ll learn how to set up a connector. Connectors receive extracted data from Empower, then transmit that data from the agent to the Tetra Data Platform, where it is placed in the data lake and processed by a pipeline. Pipelines are discussed in more detail in the next step.
Although this step primarily addresses the User Defined Integration (UDI), there are several ways to extract data from Empower and send them to the Tetra Data Lake. Two commonly-used ways are to use the User Defined Integration (UDI) and Generic Data Connector (GDC).
To determine which type of connector to use, you’ll need to know two things:
- Is the agent installed on a machine that has internet access?
- Are the files to be transferred to the data lake large (more than 500 MB) or small (less than 500 MB)?
*Connector Decision Table
|Agent Internet Access||Small Files (less than 500 MB)||Large Files (more than 500 MB)|
|Agent HAS Internet Access||UDI (API upload)||UDI (S3 upload)|
|Agent DOES NOT HAVE Internet Access||GDC/DataHub (API Upload)||GDC/DataHub (S3 Upload) with the L7 Proxy Connector|
In this Quick Start Guide, we will set up User Defined Integration that will use the S3 upload, which is the recommended method. If you want to learn more about GDC or DataHub, check out the documentation listed at the bottom of this page.
To log into the TetraScience Data Platform (TDP) complete the following steps.
- Open a supported browser and go to the Tetra Data Platform site.
- Enter your username and password, then click Sign In.
Before you proceed further, read the [Tour of the TPD](https://developers.tetrascience.com/docs/basic-tetrascience-data-platform-administration) section so that you become familiar with the basic interface.
To set up the UDI, complete the following steps.
- Click the Main Menu button, select Data Sources, then User Defined Integration.
- In the User Defined Integrations page, click Add User Defined Integration in the upper right corner of the page.
- Type the Integration Name, then click Create.
- The UDI you just named appears in the ist of User Defined Integrations. Click Add Agent.
- The Add a User Defined Agent page appears. There are three sections: Source Type, Name and Description, and Attributes. For the Source Type, select Empower from the drop down menu. (Note that the Agent ID has already been created.) Make note of the Agent ID; you'll need it when you set up the Empower Agent later in these instructions. When finished, click the Next button.
- The Name and Description section appears. Add a name and a description, then click Next.
- The Attributes section appears. You can add labels, metadata, and tags here. For this quickstart, let's add a metadata item named "Quick Start" to the files.
The differences among each of these options are addressed in Basic Concepts: Labels, Metadata, and Tags.
a. Click Advanced Fields to display the Add Metadata and Add Tags options.
b. Click Add Metadata.
c. Click the drop down, then type "Quick Start" (without the quotes).
d. A message appears: "Create: Quick Start". Click it to create the metadata tag.
e. Add the metadata value "True".
- Click Finish. The Agent is added to the UDI list.
You'll need to add a service user's account so that you can obtain a JSON Web Token (JWT), which you will need when you set up the Empower Agent in a later step.
- Click the profile icon at the top right corner of the page.
- Click Account.
- On the left side of the screen, select the Organization option.
- Click the Add User button in the Organization details page.
- In the Add a New Service User page, add the service user's name and assign the user to a role.
- Click the Add button to add the service user.
You can generate a JWT Token for a service account, regardless of role. To generate the token, complete the following steps.
- In the Organizational Details page, select Generate Token from the menu options that appear at the right side of the service user account entry.
- In the Create New Token page enter the number of days the token should be valid (1 - 180 days), then click the Generate Token button.
- The token appears in the next window. Click the Copy Token button. The token is now on your clipboard and is ready for you to paste it where it is needed. Put it in a safe place; you'll need it when you set up the Empower Agent in a later step.
- Click the Close button.
At the end of this step, you should have set up a UDI, created a service user, and generated a JWT Token.
For more information on the topics discussed in this page, see the following documentation:
Updated 4 months ago
|EDSL Quick Start - Step 2: Set Up a Pipeline|