Azure Data Factory, a tool in Azure helps Orchestrate and control the data between different systems. Data Factory consists of Pipelines and other sets of elements associated with it like Integration Runtime, etc. Once the Pipeline is designed, it is important to monitor its execution.

There are multiple ways to monitor the operations and Data Factory V2 comes with additional information like Integration runtimes, Triggers, etc., and are simplified into simple User Interface.

 

Monitoring Dashboard

The Monitoring Dashboard can be accessed through the Data Factory Console. The Author and Monitor tile in the Data Factory Blade open the Data Factory console.

Monitoring and Analyzing Pipeline and Operations in Azure Data Factory

 

The Data Factory Blade also shows the different count of execution and status along with Integration runtime resource consumption.

Monitoring and Analyzing Pipeline and Operations in Azure Data Factory    Monitoring and Analyzing Pipeline and Operations in Azure Data Factory

In the Data Factory console, the monitoring tab can be selected using the Monitoring and Analyzing Pipeline and Operations in Azure Data Factory. The monitoring Dashboard shows different views for Pipeline runs, Integration runtime and Trigger runs.

 

Pipeline Executions

The Pipelines execution can be filtered based on Time and Time zones. Each Pipeline execution shows additional details such as duration, status, parameters, and the error message if it had been failed.

Monitoring and Analyzing Pipeline and Operations in Azure Data Factory

 

The execution can be re-run and it also provides information about the activity for each execution.

Monitoring and Analyzing Pipeline and Operations in Azure Data Factory

 

Integration Runtimes

The Integration Runtimes show information about the Integration Runtimes that are configured and also the status & type for the same.

Monitoring and Analyzing Pipeline and Operations in Azure Data Factory

 

The Pipeline activities for each of the Integration Runtimes can also be monitored.

Monitoring and Analyzing Pipeline and Operations in Azure Data Factory

 

Trigger Runs

The Trigger runs show the Pipeline execution information for each Trigger. It shows information about the Trigger such as Trigger Name, Status, Trigger type and the Pipeline which is associated with it. The View is like Pipeline execution.

 

Alerts

The alerts help notify based on various signals for Azure Data Factory. Alerts for Data Factory are configured as part of the Azure Monitor Alerts.

Monitoring and Analyzing Pipeline and Operations in Azure Data Factory

 

Alerts can be used to notify on the action or act upon. Various actions such as Azure functions, Logic Apps, Webhooks, etc. and can be used to automate the post-execution action such as performing operations on the Data.

Monitoring and Analyzing Pipeline and Operations in Azure Data Factory

 

Metrics

Different metrics about the Data Factory Pipelines can be visualized from the Monitoring Metrics in Azure. Multiple metrics can be compared to build a customized view and it can be exported into Excel for advanced analysis. Various metrics such as Pipeline run metrics, Integration Runtime usage, etc. can also be measured.

Monitoring and Analyzing Pipeline and Operations in Azure Data Factory

 

Monitoring is an important part of the Azure Data Factory Pipeline. Proper alerts and metrics can help in the smooth operation of the Pipelines.

Subscribe to our Newsletter

5920 Windhaven Pkwy
Plano TX 75093.

+1 888-227-2794

+1 972-232-2233

+1 888-227-7192

solutions@visualbi.com