To determine which type of connector to use, you’ll need to know two things:
- Is the agent installed on a machine that has internet access?
- Are the files to be transferred to the data lake large (more than 500 MB) or small (less than 500 MB)?
*Connector Decision Table
Agent Internet Access
Small Files (less than 500 mb)
Large Files (more than 500 mb)
Agent HAS Internet Access
CDC (API upload)
CDC (S3 upload)
Agent DOES NOT HAVE Internet Access
GDC/DataHub (API Upload)
GDC/DataHub (S3 Upload) with the L7 Proxy Connector
- If the agent has internet access, and files are small, use the Cloud Data Connector (CDC) connector.
- If the agent has internet access, and the files are large, use CDC, and make sure that you have the S3 upload configuration enabled.
- If the agent does not have internet access, and the files are small, use the Generic Data Connector (GDC), which should be on the Data Hub.
- If the agent does not have internet access, and the files are large, use the GDC, which should be on the Data Hub. You will also need to use the L7 Proxy Connector. The L7 Proxy Connector (L7C) is a simple proxy server that uploads files directly to the data lake. L7C accommodates large files.
Updated 2 months ago