Release date: 26 July 2023

🚧

IMPORTANT

For the Tetra Empower Agent v5.1.0 and Tetra Data Schema v15 release, all of the associated Tetra Data components must be updated together. These components include the intermediate data schema (IDS), protocol, task script, and agent. Any prior versions of the IDS (v13 and earlier) will be incompatible with Tetra Empower Agent v5.1.0 and higher.

What's New

TetraScience has released its next version of the Tetra Empower Agent, version 5.1.0 as well as the Tetra Data Schema, version 15.0.0 (including task script v14.0.0 and empower-raw-to-ids Protocol v8.0.0), and Empower-message-center Protocol, version 2.1.0. These components must be installed and used together, because the components are compatible with each other only. Using any of these components with other versions of another Tetra Data component will result in a breaking change.

📘

NOTE

Tetra Data Schema v15 is being released to address a bug identified in the recently released Tetra Data Schema v14. Tetra Data Schema v15 is now the recommended version compatible with Empower Agent v5.1.0. All changes made in v14 apply to v15 and are outlined in this release note. For more information, see the Bug Fixes section.

The following are new functionalities and enhancements introduced in the Tetra Empower Agent v5.1.0 and Tetra Data Schema v15.0.0 release:

• New fields were added for the following:
  • Calibration
  • Processing Method
  • Sample Set Method
  • Project
  • Injection Status
  • Injection Sample Weight
  • Injection Auto Additions
• Support was added for bidirectional communication between Empower and laboratory information systems (LIMS) and electronic lab notebooks (ELNs) through the Tetra Data Platform (TDP).
• A new feature was added to prioritize new injections over historical injections. This feature minimizes delays of new injections during re-upload, which is required when updating existing versions of the Tetra Empower Agent.
• Performance improvements were made to the injection scan process and scan retry logic.
• Improvements were also made for GxP compliance purposes, including the following:
  • Storing culture context settings
  • Preserving data precision
  • Storing date fields in ISO format
  • Logging and auditing any Agent configuration changes
• Support was added for 3D MS data precision by adding settings to control which toolkit method and how many generation processes are used.

Prerequisites

📘

NOTE

Tetra Empower Agent release notes will now include the recommended protocol version (task script and IDS) that is compatible with each version of the Empower Agent.

The following Tetra Data components are required for the Tetra Empower Agent v5.1.0 and Tetra Data Schema v15.0.0 release:

• Tetra Data
  • lcuv-empower IDS v15.0.0
    • empower-raw-to-ids Protocol v8.0.0
    • Task Script v14.0.0
• empower-message-center Protocol v2.1.0
• Tetra Empower Agent v5.1.0

New Functionality

New Functionality includes features not previously available in the TDP or Tetra Empower Agent. These features may or may not affect "intended use" for GxP validation purposes.

Tetra Data

LCUV-Empower

The following customer-requested fields were added to the Icuv-empower IDS v15.0.0:

• runs[*].injection.status
• results[*].sample.weight
• runs[*].injection.auto_additions
• methods[*].channels[*].type
• methods[*].channels
• methods[*].channels[*].processing_locked
• New fields for methods[*].sample_set.method
• New fields for methods[*].processing[*]
• New fields for methods[*].calibration[*]
• New fields for project
• For the numeric string in RAW JSON, raw_value and value properties were added to preserve the raw value as a string in the raw_value property and to convert it to numeric in the value property. Note: This change affects 118 fields.

Tetra Empower Agent

• All fields (except data cubes) in the Agent generated RAW JSON are persisted as strings. The decimal precision and numeric separators are preserved as present in Empower and available in the toolkit. This change doesn't add any data transformation.
• The following date/time fields are persisted in RAW JSON in ISO format in the UTC time zone to remove any ambiguity in date/time format and time zone:
  • Injection
    • DateAcquired
    • SampleSetStartDate
    • SampleSetFinishDate
    • SystemCreationDate
    • Datetime Custom Fields
  • Result
    • DateAcquired
    • DateProcessed
    • ResultSetDate
    • SampleSetStartDate
    • SampleSetFinishDate
    • SystemCreationDate
    • Datetime Custom Fields
  • Channel
    • DateAcquired
    • SampleSetStartDate
    • SampleSetFinishDate
    • SystemCreationDate
  • SignOff
    • SignOffDate
    • SignOffMethodDate
  • Method
    • SampleSet.SampleSet_Method.Dated
    • Injection.InstrumentMethod.Dated
    • Result.ProcessingMethods[].Dated
  • SampleSet
    • SampleSetStartDate
    • SampleSetFinishDate
    • SystemCreationDate
    • Datetime Custom Fields
  • DbProject
    • CreateDate
    • ArchiveDate
  • Project/System Audit Trail
    • ChangeDate
  • Message Center
    • Time (prior to SR3)
    • Date (post SR3)
• To enable extracting dates in ISO format (described above), the management console now includes additional fields in the Empower Database section to define the Oracle Host Name, Service Name, and Port.
• The RAW JSON schema version, Agent version, Empower client culture settings, and Agent host machine culture settings are persisted in RAW JSON to provide the context of the data and proper data transformation in downstream processing. The RAW JSON schema version is updated to 2.1.
• All Agent configuration changes as well as the starting and stopping of the Agent, will be audited and can be viewed on the platform.
• An option was added to configure Continue Generation on Empower Toolkit Error. When set to Yes, this option will continue to generate data and log any errors from the Empower Toolkit, versus a stopping failure. If set to No, the Agent will not generate the data (injection) and log the Empower Toolkit error.
• Injections with 3D MS data can now be generated by using GetSpectraand Get3D methods, or customers can choose to generate the injections without including the 3D MS data cubes. For every project, the 3D MS generation method (GetSpectra, Get3D, or None) can be selected. The default value is Get3D (for backward compatibility for existing, selected projects). When an injection belonging to a project is generated by the agent, the specified 3D MS generation method configured for that project is added to the RAW JSON as Amazon Simple Storage Service (Amazon S3) metadata.
• New injections can be prioritized to be scanned and generated earlier than other, previously processed injections by using the Prioritize Injections created/modified in last (hours) setting. Customers can also specify advanced settings to control how many processes will be used to generate high-priority injections, normal-priority injections, and injections that have 3D MS data, which are generated by using the GetSpectra toolkit method. The setting is available in the Injection Generation Boundary Parameters section. 2 new columns have been added to Project grid of the management console:
  • Priority—shows the scan priority assigned to the project, based on if any injections have been created/modified in the hours specified for prioritizing injections. The values are High and Normal.
  • Last Scanned—shows the date/time stamp of when the project was last scanned by the agent.
• A new Advanced section was added to Agent Run Time section of the management console. Several advanced settings can be tweaked through these, including the following:
  • New settings were added to specify the number of processors to be used for normal priority injections (default is 2), new/high priority injections (default is 1), and 3D MS GetSpectra method injection (default is 1) generation. These three new settings replace the generation processor number setting available in previous Tetra Empower Agent versions.
  • The Interval of Scan (minutes) setting has been added to the Advanced popup. Its behavior is the same as in previous agents and is used to define how often the agent periodically scans Empower for new or modified injections.
  • A new setting was added to specify max upload parallelism that the agent can use (default value is total virtual processors - 1).
  • New settings have been added for injection generation max timeout (for all injections except 3D MS with GetSpectra) and injection GetSpectra generation max timeout (for injections having 3D MS data cubes and generated by using GetSpectra Toolkit method). Default values are 60 and 180 minutes, respectively.
  • A toggle was added to specify agent behavior when the scan process encounters an Empower Toolkit 105 error. The default value is Yes, and if the agent gets a 105 error for a project during the scan, it will continue the scan process (instead of getting stuck) with the next project. The error will be logged and visible in the Project grid of the management console. The project which has the 105 error will be added back to the scan queue.
• A new Auto Enable New Child option has been added to the Project grid of the management console. When this option is selected, when any new child project is created in Empower for a project that is already selected in the Agent, the new child project will be automatically enabled in the Agent. The child project will inherit the Sig Off result Option and 3D MS Method settings from its immediate parent. If the Auto Enable New Project option is selected, it will take precedence over the Auto Enable New Child option. In that scenario, all new projects added to Empower will be enabled in the agent if the Empower user that's used in the agent can scan them. The 3D MS Method for the auto-enabled project will be set to Get3D.
• A new Receive Commands toggle is available in the management console. Selecting Yes (and enabling the queue in TDP for the agent) will allow the TDP and the agent to use TetraScience APIs. By default, this toggle is set to No.
• Bidirectional APIs are now available in the agent. An existing Sample Set Method can be extracted from Empower by using the TetraScience API. A new Sample Set Method can be created by sending the required information through the API to the agent.

Enhancements

Enhancements are modifications to existing functionality that improve performance or usability, but don't alter the function or intended use of the system.

Tetra Data

LCUV-Empower

• Top-level fields were changed to bring the IDS up to current conventions
• Added an FK / PK relationship
• 3D MS data is now written to a parquet file
  • Add datacubes_metadata to capture metadata information for all 3D MS data written to parquet
• The results[*].sample.weight.unit field was removed.
• The boolean_value from CustomField definition was removed.
• The method.instrument.setups[*].needle_wash field was changed from type ["null", "bool"] to ["null", "string"]
• The results[*].peaks[*].level field was changed from type ["null", "number"] to ["null", "string"]

Message-Center

• Support was added for Empower Agent raw data schema v2, which was introduced in Empower Agent v5.0.0
• A raw_json_schema_version field was added
• A system.agent object was added
• Updated log_entries[*].sequence from int to include both raw and parsed values in log_entries[*].sequence.raw_value and log_entries[*].sequence.value.
• Removed the time field, because it's no longer needed with the new agent update.

Tetra Empower Agent

• The project user access permission schema now contains a method type and method name for all methods for a project.
• In Injection RAW JSON, additional new fields for Calibration, Processing Method, Sample Set Method, Project, and Auto-Additions have been added.
• Injection Status and Channel Status have specific fields in RAW JSON (instead of only being present in custom fields in earlier versions). The protocol parses the values into IDS from these specific fields.
• The login process now has retries when a project cannot be accessed. After a predefined number of retries, if the project cannot be accessed, an error is logged and the agent moves on to scanning the next project.
• Rerun projects now allow customers to select one, multiple, or all projects. The list of projects which can be selected to be rerun (the agent will re-upload all injections in those projects) are all the ones that have injections and that have already been generated by the agent.
• Scan and generation processes have been made more resilient with timeouts and retries when these processes get stuck and hang in the background.
• The Conversion Status column in the Project grid in the management console has been renamed Generation Status.
• In the Processing Summary of the Injection Summary tab in the management console, Pending Generation shows all of the injections that have been scanned, but not generated. A new row shows the number of injections pending generation, which have 3D MS data cubes and that the user has selected with the GetSpectra method.
• In the error grids of the Injection Summary tab of the management console, two new columns have been added:
  • Has 3D MS —is set to yes if the injection has 3D MS data cubes.
  • 3D MS Generation Method —displays the 3D MS generation method selected for the project that the injection belongs to.
• The Upload Batch Size setting in the management console has been removed. The upload process now runs continuously and doesn't require this setting.

Bug Fixes

The following customer-reported bugs are now fixed:

Tetra Data

LCUV-Empower

• Updated methods[*].calibration[*].curves[*].average_cal_points to handle an array of objects with the following fields:
  • level
  • average.value, average.response
  • calculated_value
  • percent_deviation
  • error_of_average
  • number_of_points_in_average
• Updated custom_fields to handle enum and date
• Removed conversion of boolean value under all custom_field.value fields; removed custom_field.boolean_value fields where custom_field.value_data_type is boolean.
• Added conversion of numeric values to human readable strings for method.instrument.setups[*].needle_wash
• Fixed incorrect data cube attribute assignment for the following fields:
  • datacubes[*].measures[0].name
  • datacubes[*].dimensions[0].name
  • datacubes[*].dimensions[0].unit
• An issue where ts-task-script-empower-raw-to-ids fails when Datacube Measures.Value, Wavelength.Scale, and Time.Scale have null values (Ticket #2565).

Tetra Empower Agent

• For some historical injections, in earlier versions of the Tetra Empower Agent, the generation process didn't timeout correctly, causing the generation to hang for those injections. This issue has been fixed in Tetra Empower Agent v5.1.0 (Issue #1939 and #2031).
• For certain errors in Empower Toolkit during scanning (for example, System.InvalidCastException), in previous versions of the Tetra Empower Agent, the scan process will hang. This issue has been fixed in v5.1.0. The scan process will recover, log the error, and continue to scan the next project (Issue #2013).
• 3D MS data cubes can now be generated with correct precision by using the GetSpectra Toolkit method (Issue #1900).

Known and Possible Issues

The following are known and possible issues for the Tetra Empower Agent v5.1.0 and Tetra Data Schema v15.0.0 release:

Tetra Empower Agent

• When the agent generates RAW JSONs, the agent uses the Sample Set Method name to include the sample set method information in the RAW JSON. This is the only toolkit method available. Customers might have changed the Sample Set Method. If you're using the Sample Set Method in the RAW JSON or IDS JSON, it's recommended that customers compare the SampleSet.SampleSetMethodDate in the RAW JSON and the sample.set.method.created_at in the IDS JSON with the time of the injection to confirm which Sample Set Method is referred to.
• If there are a large number of results with an injection (in the order of several thousands), the RAW JSON generation of that injection may hang. A fix for this issue is in development and testing, and will be available in a future release.
• If the Empower project name has non-ASCII characters (for example, accents or dieresis), the order of projects displayed in the Empower project browser and agent management console may not be the same.
• Empower 3 pre SR3 versions don't have Injection or Channel status for the Agent to understand when the data acquisition from a live instrument is complete. To make sure that incomplete data isn't uploaded to the TDP when the Agent is generating data for injection acquired from a live instrument by Empower 3 pre SR3, or for an injection whose data is modified as part of processing after data acquisition, the Agent will defer the data generation for the time specified in the Max Acquisition Hours setting from the latest project audit trail.
• When a project is deleted in Empower (which the agent could access through the configured Empower user), and project/user access non-experiment data is generated, the deleted project information may still be included in the RAW JSON. A fix for this issue is in development and testing, and will be available in the next patch release.
• Sample Set Method information for an Injection through the toolkit provides the Sample Set Method name of the approved Sample Set Method version instead of the Sample Set Method ID of the version used to set up the sample set. This may or may not be the Sample Set Method version that was actually used for the sample set in the Injection.
• When the Agent fetches Channel data and gets a non-Toolkit exception such as an Out of Memory error, the Injection RAW file it generates has an empty Channel data cube.

Operating Suggestions

Tetra Empower Agent

• If you're not interacting with the agent management console frequently, it's recommended to start the agent, close the management console, and then let the agent run in the background.
• If the Empower server or client is under maintenance, it's recommended to stop the Tetra Empower Agent. Remember to restart the agent when Empower is back online. The unfinished tasks will be resumed by the agent once it is started again.
• If the Empower Oracle service is stopped, then stop the agent and restart the agent after the Oracle service is back online.

Upgrade Considerations

To upgrade the agent, see Common Tetra Agent Upgrade and Installation Information. To update the protocol, see Edit a Pipeline.

During the upgrade, keep in mind the following:

• The upgrade for the Tetra Empower Agent v5.1.0 and Tetra Data Schema v15.0.0 release is NOT backward-compatible. This means the RAW files generated by this agent cannot be processed by previous protocol versions. The agent is now also able to generate 3D MS data cubes with correct data precision.
• Performance-wise, the new protocol’s runtime will be longer when compared to the previous protocol versions because of the IDS structure change and the newly added fields. From our internal testing, customers can expect an increase in runtime (tens of seconds) for Empower data without 3D Mass Spec data. For Empower data with 3D Mass Spec data, customers can expect a longer runtime because of the newly added Parquet file creation for 3D MS data. The time increase will be dependent on the size of the 3D MS data. The larger the data, the more time it will take to generate the Parquet file.
• If data generated by previous agent versions needs to be available with the latest IDS version released with this release, customers will need to configure the protocols with empower-raw-to-ids protocol v8.0.0 and re-upload all of the previously generated injections by using Rerun Project(s) in the management console and by selecting all of the projects in the list. Keep in mind that re-uploading all of the projects may take considerable time.
• If 3D MS data precision is required, configure the 3D MS Method as GetSpectra for the specific projects in the Project grid. Start the agent, and if previously generated injections with 3D MS data cubes need to be re-uploaded, use the Rerun Project(s) option to upload the specific projects. Keep in mind that 3D MS data cubes generation using GetSpectra Toolkit method is significantly slower than generating 3D MS data cubes with the Get3D Toolkit method and will increase the generation time for each injection.
• Support for Empower-Audit-Trail and Empower-Project-User Tetra Data is currently in development and testing, and will be available in a future release (targeting Q4 2023).

Other Release Notes

To view other Tetra Agent and Tetra Data release notes, see Tetra Agent and Tetra Data Release Notes.

To view other TDP release notes, see Tetra Data Platform (TDP) Release Notes.