Data Flow is an exciting feature that provides a new, robust, easy-to-use graphical-based modelling experience for ETL requirements.
From a data modelling perspective, we would require combining and loading data from various sources. With the Data Flow builder, integration of structured and semi-structured data becomes much easier, achieved using a series of standard transformations and custom transformations based on Python script that handles complex logics.
In this blog, we can see the advantages of Data Flow, how to schedule and monitor them, and available sources for data flow in SAP Data Warehouse Cloud version 2021.10.87.
Advantages of Data Flows
1. Data Persistence
To work with large data sets like data lakes, Data Flow will be the best option as data is persisted. When executed through a Data Flow, the data will be read, transformed, and persisted into the SAP DWC target.
2. Transformation capabilities
All the standard operators present in the graphical view builder can be used in Data Flow – For, e.g., join, union, projection, and aggregation. In projection, we can choose the required fields, apply filters, and create a calculated column using the 84 built-in functions categorized into conversion, date, mathematics, etc. In addition, we can group data using an aggregation node.
3. Easy to use
The objects from the remote data sources or the tables and views in our space can be accessed and added with just a drag and drop. Any changes done in columns are propagated throughout the Data Flow without manual intervention.
The “Add table” feature generates the final output table combining different data sets into the Data Flow automatically.
Available Sources
Scheduling and Monitoring Data Flows
A Data Flow can be scheduled to run recursively on a specified frequency and be monitored for completion.
To monitor a Data Flow, we can navigate to the Data Flow Monitor which displays all the executed data flows. We can view and monitor the execution details of the scheduled data flows, with an option to interrupt the currently running execution and fetch the latest run status.
Views Vs Data Flows
Conclusion
Data flow enables businesses to perform the ETL they require and make some basic transformations in an intuitive graphical UI. Data from structured and unstructured sources can be cleansed, combined, merged using various transformations providing meaningful insights.
A combination of the Data Flow Builder and other data view builders available in SAP Data Warehouse Cloud gives us even more flexibility in how we access and transform data.
Reach out to us here today if you are interested in evaluating if SAP Data Warehouse Cloud is suitable for you.
