Selecting Which Connector Type to Use

To determine which type of connector to use, you’ll need to know two things:

  • Is the agent installed on a machine that has internet access?
  • Are the files to be transferred to the data lake large (more than 500 MB) or small (less than 500 MB)?

Connector Decision Table

Agent Internet AccessSmall Files (less than 500 mb)Large Files (more than 500 mb)
Agent HAS Internet AccessNo Connector (API upload)No Connector (S3 upload)
Agent DOES NOT HAVE Internet AccessGDC/DataHub (API Upload)GDC/DataHub (S3 Upload) with the L7 Proxy Connector
  • If the agent has internet access, and files are small, use the "No Connector" option.
  • If the agent has internet access, and the files are large, use
    "No Connector", and make sure that you have the S3 upload configuration enabled.
  • If the agent does not have internet access, and the files are small, use the Generic Data Connector (GDC), which should be on the Hub v1.
  • If the agent does not have internet access, and the files are large, use the GDC, which should be on the Hub v1. You will also need to use the L7 Proxy Connector, which is a simple proxy server that uploads large files directly to the data lake.