Tetra File-Log Agent Performance Guidelines
Tetra File-Log Agent performance depends on the Agent’s deployment environment and the amount of data being processed. This topic provides key performance metrics for the Tetra File-Log Agent and describes the factors that influence those metrics. It also provides baseline performance testing results that you can use to help optimize the Tetra File-Log Agent’s performance in your environment.
For information about monitoring a Tetra File-Log Agent’s performance, see Monitor Tetra File-Log Agent Progress and Metrics.
NOTE
For help optimizing your Tetra File-Log Agent’s performance, contact your customer success manager (CSM).
Key Performance Metrics
Tetra File-Log Agent performance is typically measured by using the following performance metrics:
- Scan frequency: How frequently the Agent scans file or folder paths.
- Scan duration: How long it takes the Agent to scan file or folder paths.
- Number of files uploaded: How many files the Agent uploads to the TDP in a given time period.
- File latency: How long it takes a file to be uploaded to the TDP after it’s created or modified.
Influencing Factors
Scan frequency is configured through the File Watcher Service. Scan duration, the number of files uploaded, and file latency are determined by many factors, including the following:
- The number of scanned paths
- Connection status between the Agent and the Tetra Data Platform (TDP)
- Available disk space on the host server
- Disk usage
- CPU usage
- Memory usage
NOTE
If file system events are supported in your environment, file latency drastically improves because the Agent doesn’t need to scan all of the files in each path to detect changes. For more information, see File Systems and Protocols.
Tetra File-Log Agent Performance Testing
TetraScience conducts performance testing of the Tetra File-Log Agent with each release. Two tests are used to simulate different scenarios: One for historical data loads, and another for continuous data scanning.
NOTE
The following tables show guidelines for how a Tetra File-Log Agent performs on a host server with four logical processors, 16 GB of memory, and a 10 Gbps network connection.
Historical Data Loads
The following test simulated configuring a Tetra File-Log Agent with new paths, or adding a large number of new files into monitored paths.
Test
- Monitored paths: 25
- Number of files scanned: 100,000 (4,000 files per path at 3 MB each)
Results
Key Performance Metric | Performance Test Result |
---|---|
Scan frequency | 19 seconds |
Scan duration | 24 seconds |
Number of files uploaded | 100,000 |
File latency | - 30 seconds for the first scanned files - 2.5 hours for the full batch of 100,000 files |
Continuous Data Scanning
The following test simulated a Tetra-File Log Agent with a large database (150 GB+) that scans more than 10 million files, which are generated consistently over a three-day time period.
Test
- Monitored paths: 100
- Number of files scanned: 10 million (17 generated every minute)
Results
Key Performance Metric | Performance Test Result |
---|---|
Scan frequency | 20 minutes |
Scan duration | 20 minutes |
Number of files uploaded | 70,000+ |
(Mean) File latency | 39.5 seconds |
Updated 10 months ago