Partitioned Aggregation

Partitioning can increase processing throughput and speed of data processing, by breaking operations into multiple pieces, or partitions, allowing data processing to split across multiple hosts, and across multiple threads per host. Aggregation is one of the areas in IdentityIQ where partitioning can be used to improve performance.

Partitioned aggregation can occur at either the application level or the task level.

Some connector types support partitioning at the application level. To use partitioning for account and account group aggregation, you must configure the application for partitioning, and enable partitioning when defining an account aggregation or identity refresh task.

Application-level partitioning requires connector support and some use of partitioning statements to obtain mutually-exclusive data sets for parallel processing. This creates multiple connections to a target system and also spreads aggregation processing across the task servers.

Partitioning is supported by many but not all connectors, and can be enabled as part of the application's configuration. The way partitioning is configured varies by connector. For the most current information about a particular connector's partitioning support, refer to the IdentityIQ Connectors documentation on SailPoint's documentation portal.

Task-level partitioning is an alternative when application-level partitioning is not available or is undesirable. This variation pulls data from the target system into IdentityIQ using a single connection and multi-threads the processing across the task servers. The best practice is to use application-level partitioning whenever possible, as it has superior throughput potential when compared to task-level partitioning or traditional single-threaded aggregation, which uses a single thread for both data source connection and data processing within IdentityIQ.

Activating partitioning on an aggregation task only requires selecting the Enable Partitioning option in the task definition user interface page. This must be enabled for each aggregation task which will use partitioning, as this setting is disabled by default.

In addition, you can configure the number of objects per partition. This option sets the maximum number of records to include in each partition. IdentityIQ divides the accounts from the data source into as many partitions as required to create mutually-exclusive segments, with each containing no more than the specified number of accounts.

Only some tasks support partitioning: account aggregation, account group aggregation, identity refresh, identity request maintenance, propagate role changes, and system maintenance.