Guidelines on Volumes and Limits
Workflow Import performance is optimized by designing a Workflow structure that considers the data volumes of the total number of source records and the resulting total number of summarized target records. The transformed source records are summarized in the StageSummaryTargetData table. The volumes can be managed by structuring the data source loads across Workflows, gaining advantages in parallel processing.
| Limit | Description |
|---|---|
| Row Limit Per Workflow | 24 million summarized records |
| Best-Practice Recommendation | 1 million summarized records |
| Example | Description |
|---|---|
| Single Workflow Results | 24 million summarized records |
| Best-Practice Solution | Parse the file to be used across 24 partitioned Workflows |
The benefits of efficient Workflow structure using partitioning when working with large data sources are:
-
Performance gains through parallel processing
-
Shorter processing times
-
Faster mapping and error correction
-
More transparent data validation
Standard Workflow Record Analysis
OneStream Task Activity / Load Cube presents the details of the total number of summary records in the SelectStageSummaryTargetRows entry.
Direct Workflow Record Analysis
The Direct Load Status / Execution Status screen displays the Summary Row Count for each load process to display the total number of summary records generated.
Blend Record Analysis
The Blend process differs from other Stage Import processes by generating additional data records, or rows, rather than one-to-one or summarizations. This is because the Analytic Blend designs require the generation of aggregation points, which add to the source rows. Detailed analysis of the BI Blend Processing Logs, Live Row Count statistics, and Task Activity BI Blend Load and Transform help guide the requirements for the application and systems. Refer to the BI Blend Design and Reference Guide for more detail.