Tetra Empower Agent v5.2.x and Tetra Data Schema Release Notes
The following are the Tetra Empower Agent release notes for versions 5.2.x.
IMPORTANT
For the Tetra Empower Agent v5.2.x releases, all of the associated Tetra Data components must be updated together. These components include the intermediate data schema (IDS), protocol, task script, and agent. Any prior versions of the IDS (v13 and earlier) will be incompatible with Tetra Empower Agent v5.1.0 and higher.
v5.2.3
Release date: 2 August 2024
What's New
TetraScience has released its next version of the Tetra Empower Agent, version 5.2.3. This release focuses on resolving several known issues. There are no new functionalities in this release.
Prerequisites
The following Tetra Data components are required for the Tetra Empower Agent v5.2.3:
NOTE
Documentation for Tetra Data components is in the TetraConnect Hub. For access, see Access the TetraConnect Hub.
- Tetra Data
- LC-UV Empower IDS (lcuv-empower) v16.0.1 (recommended) or v15.0.1
empower-raw-to-ids
protocol v9.1.2 (recommended) or v8.2.1- Task script v15.1.2 (recommended) or v14.2.1
- LC-UV Empower IDS (lcuv-empower) v16.0.1 (recommended) or v15.0.1
The following Tetra Data components are compatible with Tetra Empower Agent v5.2.3:
- Empower Audit Trail IDS (empower-audit-trail) v3.0.0
empower-audit-trail-raw-to-ids
protocol v3.0.0- Task script v3.0.0
- Empower Project User IDS (empower-project-user) v2.0.0
empower-project-user-raw-to-ids
protocol v2.1.0- Task script v2.1.0
- Empower Message Center IDS (empower-message-center) v2.0.0
empower-message-center-raw-to-ids
protocol v2.2.0- Task script v2.2.0
New Functionality
New Functionality includes features not previously available in the Tetra Empower Agent. These features may or may not affect Intended Use for GxP validation purposes.
- There is no new functionality in this release.
Enhancements
Enhancements are modifications to existing functionality that improve performance or usability, but don't alter the function or intended use of the system.
Tetra Empower Agent Enhancements
- Retrieving Project Methods when generating Project/User Access Permission data is now optional and configurable through the Extract Project/User Access Permission setting in the Agent Managment Console. The setting is activated by default and set to Yes.
Bug Fixes
- This release removes the redundant
"internal"
jobs from the Tetra Empower Agent Windows Service application TetraScience.Agent.empower.Host.v5.2.3$<agent_id>
Deprecated Features
There are no new deprecated features in this release. For more information about Tetra Data Platform (TDP) deprecations, see Tetra Product Deprecation Notices.
Known and Possible Issues
- When the Agent generates RAW JSONs, the agent uses the
Sample Set Method
name to include the sample set method information in the RAW JSON. This is the only toolkit method available. Customers might have changed theSample Set Method
. If you're using theSample Set Method
in the RAW JSON or IDS JSON, it's recommended that customers compare theSampleSet.SampleSetMethodDate
in the RAW JSON and thesample.set.method.created_at
in the IDS JSON with the time of the injection to confirm whichSample Set Method
is referred to. - If the Empower project name has non-ASCII characters (for example, accents or dieresis), the order of projects displayed in the Empower project browser and Agent Management Console may not be the same.
Sample Set Method
information for an Injection through the toolkit provides theSample Set Method
name of the approvedSample Set Method
version instead of theSample Set Method ID
of the version used to set up the sample set. This may or may not be theSample Set Method
version that was actually used for the sample set in the Injection.- The Agent will not ingest the file if the injection contains the results processed using GPC/GPCV (Gel Permeation Chromatography) processing methods.
Upgrade Considerations
After customers complete the upgrade to Tetra Empower Agent v5.2.3 and open the Agent Management Console for the first time, the console prompts them to run a database upgrade script. This script is required to clean up potential duplicate injections.
To upgrade the agent, see Common Tetra Agent Upgrade and Installation Information.
v5.2.2
Release date: 1 April 2024 (Last updated: 25 June 2024)
What's New
TetraScience has released its next version of the Tetra Empower Agent, version 5.2.2. This release focuses on resolving several known issues and provides new functionalities for its associated Tetra Data components.
Prerequisites
The following Tetra Data components are required for the Tetra Empower Agent v5.2.2:
- Tetra Data
lcuv-empower
IDS v16.0.1 (recommended) or v15.0.0empower-raw-to-ids
protocol v9.1.2 (recommended) or v8.2.0- Task script v15.1.2 (recommended) or v14.1.0
The following Tetra Data components are compatible with Tetra Empower Agent v5.2.2:
empower-audit-trail
IDS v3.0.0empower-audit-trail-raw-to-ids
protocol v3.0.0- Task script v3.0.0
empower-project-user
IDS v2.0.0empower-project-user-raw-to-ids
protocol v2.1.0- Task script v2.1.0
empower-message-center
IDS v2.0.0empower-message-center-raw-to-ids
protocol v2.2.0- Task script v2.2.0
New Functionality
New Functionality includes features not previously available in the Tetra Empower Agent. These features may or may not affect Intended Use for GxP validation purposes.
Tetra Data
LCUV-Empower
- Added a
save_all_datacubes_to_parquet
config parameter to control if either all or only 3D MS datacubes are output to Apache Parquet filesNOTE
There is a limitation on the size of the data that can be saved to the Parquet file. During testing, files of approximately 700 MB have failed to generate Parquet files when the
saveAllDatacubesToParquet
value is set totrue
. If you are processing a large file, it's recommended that you set this config parameter tofalse
to avoid Parquet file generation failure. - Added parsing of top level
sample_sets[*]
to capture all sample sets (Process only
andacquired
) in a given raw JSON input - Added warning when the
datacubes[*].measures[*].unit
parameter is null - Updated parsing of
methods[*].processing.update_pet_type
tomethods[*].processing.update_ret_type
- Updated
elasticsearch.json
to include the nested fieldmethods[*].calibration.average_cal_points[*]
Enhancements
Enhancements are modifications to existing functionality that improve performance or usability, but don't alter the function or intended use of the system.
- There are no new enhancements in this release.
Bug Fixes
- Performance of the Injection Summary tab has been improved. Projects are now selected using a dialog and supports tens of thousands of projects and the injection summary and error grids now support millions of injections.
- Deferred, high priority injections will not block the generation of normal priority injections.
- Fix for issue causing multiple scans of the same project to occur in separate processes in parallel. This potentially resulted in duplicate records in the agent database for some injections.
!QuickSet
is recognized as a validSample Set Method
. The Agent can process the injection even when the optionContinue Generation on ToolKit Error
is set toNo
.- To avoid unnecessary generation errors, the Agent no longer fetches the calibration when the calibration ID is
0
.
Deprecated Features
There are no new deprecated features in this release. For more information about Tetra Data Platform (TDP) deprecations, see Tetra Product Deprecation Notices.
Known and Possible Issues
- When the Agent generates RAW JSONs, the agent uses the
Sample Set Method
name to include the sample set method information in the RAW JSON. This is the only toolkit method available. Customers might have changed theSample Set Method
. If you're using theSample Set Method
in the RAW JSON or IDS JSON, it's recommended that customers compare theSampleSet.SampleSetMethodDate
in the RAW JSON and thesample.set.method.created_at
in the IDS JSON with the time of the injection to confirm whichSample Set Method
is referred to. - If the Empower project name has non-ASCII characters (for example, accents or dieresis), the order of projects displayed in the Empower project browser and agent management console may not be the same.
Sample Set Method
information for an Injection through the toolkit provides theSample Set Method
name of the approvedSample Set Method
version instead of theSample Set Method ID
of the version used to set up the sample set. This may or may not be theSample Set Method
version that was actually used for the sample set in the Injection.- The Agent will not ingest the file if the injection contains the results processed using GPC/GPCV (Gel Permeation Chromatography) processing methods.
- Some of the Agent's internal "jobs" are duplicated in the two Windows Services applications that run within the Agent. These duplicate jobs can sometimes lead to the following issues, which don't impact data integrity:
- Additional error messages may be logged when uploading Agent logs and system metrics, because two processess are trying to upload the same log files (for example, Failed to upload Agent Log Files or The process cannot access the file ).
- Agent logs and system metrics can be uploaded twice.
- When the S3 Direct Upload option turns on, the SQLite database can be uploaded twice daily.
- Agent user audit trails can be uploaded twice to the TDP.
- When using any command, two identical messages indicating command success or failure appear in the TDP.
- When using the
CreateSampleSetMethod
command, the sameSample Set Method
is updated twice in Empower.
Upgrade Considerations
After customers complete the upgrade to Tetra Empower Agent v5.2.2 and open the management console for the first time, the console prompts them to run a database upgrade script. This script is required to clean up potential duplicate injections.
To upgrade the agent, see Common Tetra Agent Upgrade and Installation Information.
v5.2.1
Release date: 12 January 2024 (Last updated: 18 June 2024)
What's New
TetraScience has released its next version of the Tetra Empower Agent, version 5.2.1. This release focuses on resolving several known issues. There are no new functionalities or enhancements in this release.
Prerequisites
The following Tetra Data components are required for the Tetra Empower Agent v5.2.1:
- Tetra Data
lcuv-empower
IDS v16.0.1 (recommended) or v15.0.0empower-raw-to-ids
protocol v9.1.2 (recommended) or v8.2.0- task script v15.1.2 (recommended) or v14.1.0
The following Tetra Data components are compatible with Tetra Empower Agent v5.2.1:
empower-audit-trail
IDS v3.0.0empower-audit-trail-raw-to-ids
protocol v3.0.0- task script v3.0.0
empower-project-user
IDS v2.0.0empower-project-user-raw-to-ids
protocol v2.1.0- task script v2.1.0
empower-message-center
IDS v2.0.0empower-message-center-raw-to-ids
protocol v2.2.0- task script v2.2.0
New Functionality
New Functionality includes features not previously available in the Tetra Empower Agent. These features may or may not affect Intended Use for GxP validation purposes.
- There is no new functionality in this release.
Enhancements
Enhancements are modifications to existing functionality that improve performance or usability, but don't alter the function or intended use of the system.
- There are no new enhancements in this release.
Bug Fixes
- When the Agent fetches Channel data and gets a non-Toolkit exception such as an Out of Memory error, the Injection RAW file generation now fails rather than creating an empty
Channel
data cube. - Improvements to how the Agent generates data cubes reduces the chances of Out of Memory errors during Injection generation.
Deprecated Features
There are no new deprecated features in this release. For more information about Tetra Data Platform (TDP) deprecations, see Tetra Product Deprecation Notices.
Known and Possible Issues
- When the agent generates RAW JSONs, the agent uses the
Sample Set Method
name to include the sample set method information in the RAW JSON. This is the only toolkit method available. Customers might have changed theSample Set Method
. If you're using theSample Set Method
in the RAW JSON or IDS JSON, it's recommended that customers compare theSampleSet.SampleSetMethodDate
in the RAW JSON and thesample.set.method.created_at
in the IDS JSON with the time of the injection to confirm whichSample Set Method
is referred to. - If the Empower project name has non-ASCII characters (for example, accents or dieresis), the order of projects displayed in the Empower project browser and agent management console may not be the same.
Sample Set Method
information for an Injection through the toolkit provides theSample Set Method
name of the approvedSample Set Method
version instead of theSample Set Method ID
of the version used to set up the sample set. This may or may not be theSample Set Method
version that was actually used for the sample set in the Injection.
Upgrade Considerations
After customers complete the upgrade to Tetra Empower Agent v5.2.1 and open the management console for the first time, the console prompts them to run a database upgrade script. This script is required to implement the improvements in data cube generation.
To upgrade the agent, see Common Tetra Agent Upgrade and Installation Information.
v5.2.0
Release date: 20 December 2023 (Last updated: 18 June 2024)
IMPORTANT
For the Tetra Empower Agent v5.2.0 release, all of the associated Tetra Data components must be updated together. These components include the intermediate data schema (IDS), protocol, task script, and agent. Any prior versions of the IDS (v13 and earlier) will be incompatible with Tetra Empower Agent v5.1.0 and higher.
TetraScience has released its next version of the Tetra Empower Agent, version 5.2.0. This release focuses on reducing data latency and improving the Agent’s performance and usability by introducing new functionalities and enhancements, such as the following:
- Multiple Agents can now connect to the same Empower Server, which allows the system to scale horizontally
- Ability to set Scanning Priority for projects
- Shorter delays when the Agent loses connectivity
- Ability to configure the maximum number of projects included during the scan process (set with a default value of 20 that helps optimize performance and reliability)
This release also includes several bug fixes for scanning, injection generation, and project-user access permissions.
Here are the details for what’s new in Tetra Empower Agent v5.2.0.
Prerequisites
NOTE
Tetra Empower Agent release notes will now include the recommended protocol version (task script and IDS) that is compatible with each version of the Empower Agent.
The following Tetra Data components are required for the Tetra Empower Agent v5.2.0:
- Tetra Data
lcuv-empower
IDS v16.0.1 (recommended) or v15.0.0empower-raw-to-ids
protocol v9.1.2 (recommended) or v8.2.0- Task script v15.1.2 (recommended) or v14.1.0
The following Tetra Data components are compatible with Tetra Empower Agent v5.2.0:
empower-audit-trail
IDS v3.0.0empower-audit-trail-raw-to-ids
protocol v3.0.0- Task script v3.0.0
empower-project-user
IDS v2.0.0empower-project-user-raw-to-ids
protocol v2.1.0- Task script v2.1.0
empower-message-center
IDS v2.0.0empower-message-center-raw-to-ids
protocol v2.2.0- Task script v2.2.0
New Functionality
New Functionality includes features not previously available in the TDP or Tetra Empower Agent. These features may or may not affect "intended use" for GxP validation purposes.
Tetra Data
LCUV-Empower
- Added config parameter
save_all_datacubes_to_parquet
to control whether to output all or only 3D MS datacubes in a parquet file- There is a limitation on the size of the data to be saved to the Parquet file. Based on our experience, files of approximately 700MB have failed to generate Parquet files when the saveAllDatacubesToParquet configuration is set to true. Therefore, if you are processing a large file, we recommend setting saveAllDatacubesToParquet to false to reduce the data size that goes into the Parquet file
- Added parsing of top level
sample_sets[*]
to capture all sample sets (Process only
andacquired
) in a given raw JSON input - Added warning when
datacubes[*].measures[*].unit
is null - Updated parsing of
methods[*].processing.update_pet_type
tomethods[*].processing.update_ret_type
- Updated elasticsearch.json to include the nested field
methods[*].calibration.average_cal_points[*]
Empower-Audit-Trail
- Removed
log_entries
as a nested field in Elasticsearch - Added support for Empower Agent v5.x and higher
Empower-Project-User
- Added support for Empower Agent v5.x and higher
- Added system.agent
Tetra Empower Agent
Multiple Agents Can Now Upload Data to the Same Location
Customers can now configure multiple Agents to upload data to the same Amazon Simple Storage Service (Amazon S3) bucket within the TDP by specifying a Destination Id
on the Connection Configuration page of the Agent Management Console.
New Scanning Priority Settings for Projects
Empower projects can now be assigned a Scanning Priority in the Agent Management Console: High or Normal. A separate process runs the High priority scans and there are fewer High priority projects thus reducing scan latency.
NOTE
Setting Scanning Priority at the project level overrides the Agent's time-based, automatic prioritization settings.
Empower Server Connectivity Is Now Monitored
The Agent’s connectivity to the Empower server is now monitored. If connectivity is lost, the Agent also now automatically terminates any scan and generation processes that are running, so they no longer need to wait for a timeout to occur first.
To deactivate this functionality, customers can turn off the Enable Empower Connection Status Check
toggle in the Advanced Settings dialog in the Agent Management Console.
Enhancements
Enhancements are modifications to existing functionality that improve performance or usability, but don't alter the function or intended use of the system.
Tetra Empower Agent
New Max Number of Projects per Scan Process Setting
A new Max Number of Projects per Scan Process
field in the Advanced Settings dialog in the Agent Management Console gives customers the ability to configure the maximum number of projects that are scanned by a single scan process. The default value is set to 20 to help optimize both performance and reliability.
NOTE
Decreasing the default
Max Number of Projects per Scan Process
value can help maintain a more strict priority ordering, if desired. Including a smaller number of projects in each scan can help reduce the risk of delaying the scan of later projects within the batch when the scan encounters an error.
Local States for the Project/User Access Permission Non-Experiment Service Can Now Be Cleared
Customers can now clear the local state for the Agent’s Project/User Access Permission Non-Experiment service. Clearing this local state causes a complete scan and generation of Project / User access permissions, rather than an incremental generation, and can be helpful when the service’s local state is no longer valid. (Issue #2854)
To clear the local state, customers can now select the Clear Cache button on the Injection Service - Non-Experiment Summary - Project/User Access Permission tab in the Agent Management Console.
Additional Enhancements
Logging improvements for errors that occur when fetching projects from Empower during generation.
Bug Fixes
The following bugs are now fixed:
Tetra Data
Empower-Audit-Trail
- Fixed parsing of long timestamp format
Tetra Empower Agent
Scanning Process Bug Fixes
- Out of memory (OOM) exceptions are now handled during scanning, and the Agent terminates the scan process, rather than trying to scan the remaining projects in each batch. The remaining projects in the batch are now added as a separate batch for later scanning.
- Unrecoverable errors are now handled during priority scanning. The Agent then terminates the scan process immediately.
- Empower Software Development Kit (SDK) errors that are unrecoverable (those error codes ending in 105) are now handled. The Agent now fetches result objects and immediately terminates the scan process. During testing, these errors have proven to persist until the process ends and a new process is launched.
- A separate, configurable timeout and batch size for project priority scanning is now available in the Advanced Settings dialog of the Agent Management Console.
Generation Process Bug Fixes
- Access to the channel data files when retrieving raw data is now validated by the Agent. Generation now fails if the data file isn't accessible, because the SDK returns without error when fetching 2D channel data and returns an empty data cube.
- Corrupted state (error code 0xc0000005) and OOM exceptions that were previously unhandled during scanning and generation are now handled and have improved logging. (Issue #2649)
- Injection results are now fetched by using paging to avoid issues when a single injection has a large number of results.
- Injections that fail generation because they exceed the generation timeout are now retried with a longer timeout.
- For Empower 3 pre SR3, the Agent now ingests the injections immediately if injection data acquisition is complete. If the data acquisition isn't complete, the Agent will ingest the injection after the maximum number of acquisition hours that are configured. Previously, the Agent always waited for the maximum number of acquisition hours before ingestion.
Project / User Access Permission Process Bug Fixes
- The Agent can now handle and recover from being in an Invalid state in the local store when the state was caused by errors generated when retrieving Project/User Access Permission non-experiment data.
NOTE
After upgrading to Tetra Empower Agent v5.2.0, any local state stored for the Project/User Access Permission non-experiment data that’s not valid will be removed so that it can be regenerated.
- Optimized Agent processing of Project / User Access Permissions to avoid unnecessary processing. The output remains unchanged.
- The non-experiment generation process timeout is now separately configurable in the Advanced Settings dialog of the Agent Management Console by using the NonExperiment console Timeout (minutes) field.
- Projects that aren't accessible by the Empower user account that the Agent is using are no longer scanned by the Project / User Access Permission scanning process. Because of this update, inaccessible projects will no longer cause error logs during scans.
User Interface Bug Fixes
The number of pending uploads reported in the Agent Management Console now displays accurately.
Deprecated Features
There are no new deprecated features in this release. For more information about TDP deprecations, see Tetra Product Deprecation Notices.
Known and Possible Issues
The following are known and possible issues that were previously identified:
Tetra Empower Agent
- When the agent generates RAW JSONs, the agent uses the
Sample Set Method
name to include the sample set method information in the RAW JSON. This is the only toolkit method available. Customers might have changed theSample Set Method
. If you're using theSample Set Method
in the RAW JSON or IDS JSON, it's recommended that customers compare theSampleSet.SampleSetMethodDate
in the RAW JSON and thesample.set.method.created_at
in the IDS JSON with the time of the injection to confirm whichSample Set Method
is referred to. - If the Empower project name has non-ASCII characters (for example, accents or dieresis), the order of projects displayed in the Empower project browser and agent management console may not be the same.
Sample Set Method
information for an Injection through the toolkit provides theSample Set Method
name of the approvedSample Set Method
version instead of theSample Set Method ID
of the version used to set up the sample set. This may or may not be theSample Set Method
version that was actually used for the sample set in the Injection.- When the Agent fetches Channel data and gets a non-Toolkit exception such as an Out of Memory error, the Injection RAW file it generates has an empty
Channel
data cube.
Operating Suggestions
- If you're not interacting with the agent management console frequently, it's recommended to start the agent, close the management console, and then let the agent run in the background.
- If the Empower server or client is under maintenance, it's recommended to stop the Tetra Empower Agent. Remember to restart the agent when Empower is back online. The unfinished tasks will be resumed by the agent once it is started again.
- If the Empower Oracle service is stopped, then stop the agent and restart the agent after the Oracle service is back online.
Upgrade Considerations
To upgrade the agent, see Common Tetra Agent Upgrade and Installation Information. To update the protocol, see Edit a Pipeline.
This version is compatible as a direct upgrade from Tetra Empower Agent v5.1.0. During the upgrade, keep in mind the following:
- The upgrade for the Tetra Empower Agent v5.1.0 and Tetra Data Schema v16.0.0 release is NOT backward-compatible. This means the RAW files generated by this agent cannot be processed by previous protocol versions. The agent is now also able to generate 3D MS data cubes with correct data precision.
- Performance-wise, the new protocol’s runtime will be longer when compared to the previous protocol versions because of the IDS structure change and the newly added fields. From our internal testing, customers can expect an increase in runtime (tens of seconds) for Empower data without 3D Mass Spec data. For Empower data with 3D Mass Spec data, customers can expect a longer runtime because of the newly added Parquet file creation for 3D MS data. The time increase will be dependent on the size of the 3D MS data. The larger the data, the more time it will take to generate the Parquet file.
- If data generated by previous agent version is not available for re-upload, empower-raw-to-ids protocol v8.2.0 allows for migration of previous agent versions to IDS v15.0.0. Only data available in the RAW JSON from the previous agent versions will be populated in the IDS. Data not available is populated as null values.
- If 3D MS data precision is required, configure the 3D MS Method as
GetSpectra
for the specific projects in the Project grid, save the changes, and start the agent. Keep in mind that 3D MS data cubes generation using GetSpectra Toolkit method is significantly slower than generating 3D MS data cubes with the Get3D Toolkit method and will increase the generation time for each injection.
Other Release Notes
To view other Tetra Agent and Tetra Data release notes, see Tetra Agent and Tetra Data Release Notes.
To view other TDP release notes, see Tetra Data Platform (TDP) Release Notes.
Updated about 2 months ago