After the Tetra Agent extracts copies of the data files from the data source, it sends those files to the Tetra Data Platform (TDP). This page shows how to configure integrations and connectors to transfer data to the TDP.
TetraScience provides several different ways for Tetra Agents to integrate with the TDP.
|Agent machine can only connect to the Tetra Hub v1 (previously Tetra Data Hub)||Agent machine can connect directly to the Cloud|
|Agent uses AWS APIs (Amazon S3, CloudWatch, etc.)||S3 via proxy|
1. Agent connects to Generic Data Connector (GDC), sends it the agent ID and JSON Web Token (JWT).
2. GDC responds with temporary credentials and the proxy location.
3. Agent uses temporary credentials and proxy to upload to S3.
|S3 without proxy|
(S3 Direct Upload option)
1. Agent connects to TDP, sends it agent ID and JSON Web Token.
2. TDP responds with temporary credentials.
3. Agent uses temporary credentials to upload to S3.
|Agent uses TetraScience's simplified wrapper||Standard|
1. Agent connects to GDC, sends it Agent ID and content.
1. Agent connects to TDP, sends it Agent ID, JWT, and content.
To determine which type of connector to use, you need to know two things:
- Is the Tetra Agent installed on a machine that has internet access?
- Are the files to be transferred to the data lake large (more than 500 MB) or small (less than 500 MB)?
|Agent Internet Access||Small Files (less than 500 mb)||Large Files (more than 500 mb)|
|Agent HAS Internet Access||TDP (API upload)||TDP (S3 upload)|
|Agent DOES NOT HAVE Internet Access||GDC/DataHub (API Upload)||GDC/DataHub (S3 Upload) with the L7 Proxy Connector|
- If the agent has internet access, use TDP (No Connector), and make sure that you have the S3 Direct Upload configuration enabled.
- If the agent has internet access, and files are small, you can use No Connector without S3 Direct Upload.
- If the agent does not have internet access, and the files are large, use the Generic Data Connector (GDC), which should be on the Hub. You will also need to use the L7 Proxy Connector (L7C). The L7C is a simple proxy server that uploads files directly to the data lake. L7C accommodates large files. The L7 Proxy Connector should also be set up on the Hub. Make sure that you have the S3 Direct Upload configuration enabled.
- If the agent does not have internet access, you cannot configure an L7C. If the files are small, you can use the Generic Data Connector (GDC) on the Hub v1 without S3 Direct Upload.
Typically, the S3 Direct Upload option is preferred—even in cases where it’s not strictly required by file size— because of its performance. When an Agent uses the TDP (No Connector) configuration with the S3 Direct Upload option activated, the TDP calculates checksum and verifies it by using AWS 128-bit MD5 checksum. For Agents not using the S3 Direct upload option, the Agent verifies the response code to determine if the upload was successful. Checksum isn't used.
- The Agent's host machine must be on Windows 10 or newer, or Windows Server 2016 for the Cloud Data Connector to connect to it properly.
After determining which connector to use, you must specify the Connection URL. The following are example GDC and TDP connection URLs:
- GDC URL example: http://10.100.1.1:8888/generic-connector/v1/agent
- TDP URL example: https://api.myorg-dev.com/v1/data-acquisition/agents
Tetra Agents can be integrated with the Tetra Data Platform (TDP) through the Tetra Hub v1 (previously Tetra Data Hub). Hub v1 is an on-premise server that facilitates securely transferring data extracted from a Tetra Agent to the TDP. The Agent connects to the Hub through the Generic Data Connector (GDC).
For more information, see Generic Data Connector.
- A Tetra Hub v1 must be set up.
- A Connector must be added to the Tetra Data Hub.
- The Hub and Connector must be online.
Because the Tetra Agent transfers data with a Tetra Hub v1 Connector, you must register the Agent first on the TDP with a specific connector.
To add a Tetra Agent to your existing Connector on the TDP, follow the instructions in Create a New Agent and choose your existing Generic Data Connector (GDC).
Besides the connection through the Data Hub/Data Connector, the Agent can directly upload the files to the TDP.
- Client Organization Slug (orgSlug) is created in the TDP.
- A user account is created in the Organization Slug
- Sign in to the TDP.
- Create an Agent entry in the TDP by follow the instructions in Create a New Agent and choose No Connector.
- After creating the Agent, choose Continue Installing Agent to configure the installed Agent with connection information and service user credentials.
To upload large files (size is larger than 500 MB) to the TDP by using the Generic Data Connector option, you must add a Tetra L7 Proxy Connector to your Hub v1.
For uploading large files (more than 500MB), make sure that you select S3 Direct Upload in the Agent and point to the L7 Proxy Connector. Without S3 Direct Upload configured on the Agent, the L7C will fail at uploading large files.
Before you can create and configure an L7C, you must do the following:
- Install and configure a Tetra Hub v1
- Install and configure a Generic Data Connector (GDC) on the Hub v1
- Make sure that the Hub v1 and GDC are online
To create an L7 Proxy Connector, following the instructions in Create an L7C.
If your Connector isn't working properly, didn't pick up the correct configuration, or if you've added an L7 Proxy Connector to your Hub, manually sync the Hub by doing the following:
- Sign in to the TDP.
- In the left navigation menu, select the hamburger menu icon. Then, choose Data Sources and select Data Hub. The Data Management page appears.
- Find the Data Hub that you want to sync. Then, select the right hamburger menu icon in that Hub's row and choose Sync. It can take up to three minutes for the Sync action to complete.
If you’ve made custom configuration changes within the Connector containers—for example, modifying an L7 proxy’s
squid.conffile—the changes will be lost during the synchronization process.
Updated 20 days ago