Monitor Pipeline File Processing
Once pipelines are set up and enabled, they can process files when triggered. There are several ways that you can monitor pipeline processing:
- View the TDP File Processing Page
- Use the Workflow Search API
- Email Notifications
If you'd like some basic information on pipelines before you start, see the Pipeline Overview page.
Use the TDP File Processing Page to Monitor the Pipeline
Pipeline processing can be easily monitored on the File Processing page. This page allows you to monitor pipeline processing, files, workflows, and logs. You can also view and download files and more.
Viewing Pipeline Processing Details and Files
- In the TDP, click the hamburger menu and select Pipelines, then File Processing.
- Pipelines are listed on the left side of the screen. To find the pipeline you want, do the following.
- To search for a specific pipeline, type the information in the search box in the upper left corner. Click the drop-down menu to toggle between searching by name or by protocol.
- Active and historical workflow counts are shown near the top of the window. The active workflow count shows all workflows that are pending or currently being processed. The historical workflow count represents workflows that have been completed or have failed. If workflows are active, you can click the refresh button next to it to view the latest status.
- Browse the pipelines on the left side of the page and click on the one you want to see. Use the scrollbar if needed.
There might be an overlap between active and historical counts during reprocessing. A previously completed fileID will have a historical record and an active record while it is being reprocessed.
- If you'd like to see a specific file, files that have certain date stamps, or that are of a certain status (All, Pending, In Progress, Completed, Failed) use the filters at the top of the page. You can also choose to simply browse, using the scrollbar as necessary.
- You can select files by clicking on them one at a time or by selecting several or all of them. To select the files, click the check box next to the File field label.
For each file, the following information is shown:
- File Name
- The number of Output Files, if any are generated
- File Status (Pending, In Progress, Completed, or Failed)
- Timestamp of the File Status
The right side of the window has two panels: Pipeline and Workflow.
- In the Pipeline Panel, you can click a link that will take you to a page where you can edit your pipeline, scan for unprocessed files, and view high-level trigger, protocol, and step details.
- In the Workflow Panel, you can click links to open workflows, files, and a file workflow history. You can also use it to reprocess files, view workflow logs, manage files, and more.
Using the Pipeline Panel to Access Pipeline Editor, Scan Files, and View Details
The following topics explain how see the Edit Pipeline page, scan for unprocessed files, selected unprocessed files for processing, and view high-level trigger, protocol, and step details.
Editing a Pipeline
You can go to the page that will allow you to edit the pipeline by clicking the Edit Pipeline button in the Pipeline Panel. Detailed documentation on how to edit pipelines appears in this topic.
Scanning Unprocessed Files
To scan for unprocessed files in the Pipeline Panel, click the Scan for Unprocessed Files button. The date of the last scan is updated.
Processing Unprocessed Files
To process files in the Pipeline Panel, complete the following steps.
- Click Select Processed Files. Files appear in a pop up window.
- Click the checkboxes to the left of the files you want to process, then click Process Selected Files. Or, click Process All Unprocessed Files to process everything that has not yet been processed.
Note that Retrying and File and Reprocessing a file are very two different terms. For more information on both, see this topic.
Retrying and Reprocessing Files
Retry and Reprocess are two related, but different terms. By default, if there is an error or issue that prevents the processing of a file, the pipeline retries processing three more times before the pipeline fails. This is done automatically, but you can do it manually as well in the TDP Dashboard. Reprocess is starting the process again; it is as if you've re-uploaded the file.
Here is a table that compares both features and explains what happens if you edit the pipeline.
|If you configure the pipeline to use another protocol.||Still uses the old protocol.||Will use the new protocol.|
|If you force overwrite the protocol (and the protocol version stays the same)||Will use the force-written protocol.||Will use the force-written protocol.|
|If the protocol refers to a wildcard version of a task-script e.g. 1.x and you upload a bumped minor version of the task-script.||Will use the new task script version.||Will use the new task script version.|
|If you force overwrite a task-script (and the script version stays the same).||Will use the force-written script.||Will use the force-written script.|
|If you update the pipeline config value directly in the pipeline design page.||Still uses the old pipeline config value.||Will use the new pipeline config value.|
|If you update a non-secret pipeline config value on Shared Settings page.||Still uses the old pipeline config.||Will use the new pipeline config.|
|If you update a secret pipeline config value on Shared Settings page.||Will use the new secret value in most cases (see note below)||Will use the new secret value.|
If you update a secret pipeline config value on the Shared Settings page, the secret is resolved when you create a new task instance, and it can only get the latest secret from SSM Parameter Store. If the workflow still uses an existing task instance, it will use the old secret. On the other hand, if the workflow creates a new task instance, it will use the new secret. The reason for this is that secret only appears in SSM or task instance, and the software doesn't copy the secret value anywhere else, so when you retry the workflow, it can only get a secret reference, not the real value.
Canceling Pending Workflows
To cancel pending workflows, in the Pipeline screen, click the Cancel all pending workflows button
Viewing Trigger, Protocol, and Step Details
In the Pipeline Panel, view the Trigger, Protocol, and Step details.
- The Trigger shows the criteria that must be met for the pipeline to process a file.
- The Protocol shows the following:
The namespace is a combination of which organization the file belongs to and who can use the file. Namespaces are discussed in detail in this topic.
The slug is the unique name of the protocol. Slugs are discussed in detail in this topic.
* The version shows the version number of the pipeline used.
- The Steps show the name of each step that is part of the pipeline workflow.
Using the Workflow Panel to View Workflows, Workflow Histories, Logs, File Properties, and to Manage Files
The Workflow panel provides details on files and workflows. It also allows you to view, download, add attributes (metadata, labels, and tags), and reprocess them. This panel becomes visible when you select one or more files from the list in the center of the page. You can also view and download files from this panel.
Viewing a Workflow
To view a workflow, complete the following steps.
- To open a workflow, select a file by clicking the checkbox next to it, then click the Open Workflow button.
- The workflow screen appears.
The following table describes the items in this window in detail.
|Input file||File input into the pipeline for processing. Clicking this will display File Versions, File Details, Workflow History and Related Files.|
|Pipeline Name||Name of the pipeline. Clicking this displays the Pipeline Manager screen that shows the pipeline name, protocol, and last configuration update.|
|Pipeline Protocol||Name of the protocol that is used in the pipeline. Clicking this displays the source file, the value for the last Commit, when it was last updated, the protocol README, and the Protocol steps for each version of the protocol. This can give you valuable information about each version of a protocol.|
|Output Files||Lists the output files. Click on a file to download it to your device.|
|Started||Lists the time the pipeline processing started.|
|Ended||Lists the time the pipeline processing ended.|
|Workflow ID||Displays an abbreviated ID for the workflow (first and last 11 characters). If you need the entire number, check the JSON Log or the JSON Tree (see information in the Attempt Log section.|
|Workflow Execution Time||Indicates how long a workflow takes to run once it is scheduled. Clicking this shows the workflow timeline.|
|Status||Status of the processing: pending, in progress, completed, failed.|
|Steps||List of steps in the pipeline.|
|Attempt Logs||Lists logs that show information about each attempt to run the step. For each attempt, three logs are shown. Basic Log shows log messages for a specific test attempt. JSON Log shows Basic Log data in JSON format with additional data such as taskId and workflow. JSON Tree shows the same data as the JSON Log, but is arranged in an expandable/collapsable tree format.|
The following show examples of the different logs.
Viewing the Workflow History
To view the file workflow history, complete the following steps.
- View the Workflow History information near the bottom of the Workflow panel. The timestamp that indicates when the workflow was completed appears.
- Click the workflow to view more details about the workflow.
View Error Logs, Workflow Status, and other Details
- To view logs, click a file on the page, then click the View Logs button in the Workflow panel. The Workflow Logs screen appears.
- The log file appears in the bottom half of the screen. By default, the log is displayed at the “info” level, which provides general information about pipeline processing.
a. If you want to troubleshoot a problem, change the log level to debug by moving the Display debug output slider to the right. More details about processing appears.
b. To view task logs, click the View Task Logs in Cloudwatch. Cloudwatch is a metrics repository in AWS that contains log data.
The following table describes the fields shown on the screen.
|Description||Short description of the pipeline.|
|Pipeline run status||Indicates the status of the pipeline run (pending, in-progress, completed, failed.)|
|Task Attempts||Indicates how many attempts, and whether the attempts are pending, in-progress, completed, or failed.|
|Display Debug Output||Indicates whether the debug output should be shown.|
Viewing File Properties
To view file properties, select a file, then look at the File Properties in the Workflow panel. It shows the Input file, its source type, and name, as well as the same information for the output file if there is one.
To view a file and download a file, complete the following steps.
- Select a file, then in the Workflow panel, click the name of the file you want to see.
- A page appears with three panels: File Versions, File Details, and Workflow History
- The File Versions panel shows the different versions of the file. The Current version has "Current" next to it.
- The File Details panel contains the File Name, File ID, Date Uploaded, File Path, Source Type, Source Name, and Size appears along with any attributes.
- The Workflow History page shows related files, including those that have been derived from this file (output).
In the File Versions panel, you can download a version of the file, preview the file, view more file information, add a new version, or add attributes.
Downloading a File
To download the file, click the Download Version button.
Previewing a File
To preview the file, click the File Preview button.
View More File Information (JSON)
To view more details by clicking the View More File Information (JSON) button to view the file in JSON format.
Uploading New File Versions
To upload a new file version, complete the following steps.
- Click "AddNew Version". The Upload a New File Version screen appears.
- If desired, add a label. Labels are explained in detail in this topic. To add a label, click the Add New Label link, then add the Label Name and Label Value. For more information on accepted values, see this topic.
- If you want to add metadata or tags, click the Advanced Fields section.
- Add the metadata and/or tags.
- To add metadata, click the Add Metadata link, then add the Metadata Field and Metadata Value. For the characters allowed, see this topic.
- To add tags, click the Add Tag checkbox and enter the tag. For the characters allowed, see this topic.
- Either click the file box and select a file using your computer's file browser or drag a file to upload.
- When complete, click the Upload button.
Add Attributes (Metadata, Labels, Tags)
To add attributes (Metadata, Labels, or Tags) to a file, complete the following steps.
1 Click the Add Attributes button. The Edit Attributes screen appears.
To add a label click the Add New Label link, then add the Label Name and Label Value. For more information on accepted values, see this topic.
To add metadata, click the Add Metadata link, then add the Metadata Field and Metadata Value. For the characters allowed, see this topic.
* To add tags, click the Add Tag checkbox and enter the tag. For the characters allowed, see this topic.
- Click Save.
Deleting a File
To download the file, click the Remove link.
Use the API to Monitor Pipeline Processing
You can use the API to monitor pipeline processing. See Search Workflows API for more details.
Use the Notifications Feature to Get Emails about the Outcome of Pipeline Processing
You can have emails sent that indicate whether a pipeline is successful or has failed.
See Set Notifications for more details.
Updated about 1 month ago