Partitioning

Partitioning is used to break operations into multiple pieces, or partitions, allowing data processing to split across multiple hosts, and across multiple threads per host. The overall goal with partitioning is to increase processing throughput and speed.

Partitioning is supported for account and account group aggregation tasks, for identity refresh tasks, for generation of manager and targeted certifications, and for role propagation.

Some connector types support partitioning at the application level. To use partitioning for account and account group aggregation, you must configure the application for partitioning, and enable partitioning when defining an account aggregation or identity refresh task.

If partitioning is enabled in an aggregation task that is acting on a connector application that does not support partitioning, partitioning will take place at the IdentityIQ database level and not at the connector level; this is referred to as generic partitioning or task-level partitioning.

Applications are configured as part of the Account Settings on the Configuration tab of the Application Configuration page. See Configuring an Application.
For task details, see Account Aggregation, Identity Request Maintenance, and Identity Refresh.
For certification details, see Manager, Application Owner, and Advanced Access Reviews and Scheduling a Targeted Certification.

Note: Partitioning is not available on all tasks or certifications. Partitioning is available for Account Aggregation, Account Group Aggregation, Identity Refresh, Perform Identity Request Maintenance, and Perform Maintenance tasks, and for Manager and Targeted Certification generation. Partitioning is also not available on all application types. Partitioning is controlled by both the configuration of the applications you are using and the configuration of the applications used to communicate with those applications.

How Partitioning Works

Each partition is placed in a global queue, and machines (or hosts) in a cluster compete to execute the partitions in the queue. Machines are added or removed from the cluster dynamically with automatic balancing. If a machine fails or is taken down while processing a partition, the partition is placed back into the queue and reassigned to a different machine.

A single result object is shared by all partitions and is continually updated so you can monitor the overall progress of the partitioned operation. When all partitions have finished executing, the result is marked complete.

Each instance of IdentityIQ includes a Server object containing information about what is happening in that instance. For machines running multiple instances of IdentityIQ, each instance must be assigned a unique iiq.hostname and have a unique Server object.

The Server objects include a heartbeat service that is updated by a new system thread on a regular basis. By monitoring server heartbeats, machines in the cluster can detect when another machine fails. When this happens any partitioned requests that were running on that machine are restarted and picked up by a different machine in the cluster, so that failure of one machine does not terminate an entire long running task.

Server objects include some statistics, such as the number of request threads currently active, and the request types that are executing. You can view the state of the machines in your cluster on the Administrator Console page. See Using the Administrator Console.