Partitioning Aggregation

The SQL Loader connector supports manual partitioning through configured SQL statements.

The Partitioning Enabled configuration parameter must be selected and the list of SQL statements must be specified in the Partitioning Statements field.

For example, if there is an employee dataset that has 100,000 rows with a sequential employeeId field, the partitioning statements that can be used are as follows:

Copy 
select x,y,z from a where employeeId <= 10000;
select x,y,z from a where employeeId > 10000 AND employeeId <= 20000;
select x,y,z from a where employeeId > 20000 AND employeeId < =30000;
…
select x,y,z from a where employeeId > 90000;

The above example has 10 partitions, handling approximately 10,000 accounts, and the last sentence (with employeeId > 90000) handles a larger number of accounts depending on the total number of employees in the system.

Note
If noAttributePromotion is set to true, the attribute promotion is skipped during aggregation.