Selecting Which Connector Type to Use
To determine which type of connector to use, you’ll need to know two things:
- Is the agent installed on a machine that has internet access?
- Are the files to be transferred to the data lake large (more than 500 MB) or small (less than 500 MB)?
Connector Decision Table
| Agent Internet Access | Small Files (less than 500 mb) | Large Files (more than 500 mb) |
|---|---|---|
| Agent HAS Internet Access | No Connector (API upload) | No Connector (S3 upload) |
| Agent DOES NOT HAVE Internet Access | GDC/Data Hub (API Upload) | GDC/Data Hub (S3 Upload) with the L7 Proxy Connector |
- If the agent has internet access, and files are small, use the "No Connector" option.
- If the agent has internet access, and the files are large, use
"No Connector", and make sure that you have the S3 upload configuration enabled. - If the agent does not have internet access, and the files are small, use the Generic Data Connector (GDC), which should be on the Data Hub.
- If the agent does not have internet access, and the files are large, use the GDC, which should be on the Data Hub. You will also need to use the L7 Proxy Connector, which is a simple proxy server that uploads large files directly to the data lake.
Updated 8 months ago