Refreshing Changed Identities Only (Delta Identity Refresh)

A "delta" identity refresh lets you update only those Identity Cubes that have changed since your last aggregation(s), rather than updating all identities. This can result in a significant reduction in refresh time, and can remove or reduce the need to partition your identities into subsets for efficient refresh processing.

In most cases, identities which have had no changes to their attributes or accounts ("link" objects) as a result of aggregations are not likely to have new policy violations or need new workflows launched to handle state changes. These identities can therefore be skipped or not processed by the Identity Refresh task. In contrast, identities that have undergone some kind of change, referred to as some kind of "delta," should be processed by the Identity Refresh task.

IdentityIQ lets you set up your tasks to refresh only the changed or delta identities; this is a two-step process:

  1. Configure and run an aggregation task to mark identities as changed when attribute or account data on the identities has been modified.

  2. Configure and run an Identity Refresh task to perform their functions only on the marked identities.

Marking Identities as Changed

During an aggregation, details of some identities are changed, while some others may not be. IdentityIQ's aggregation tasks include a setting that lets the task flag any identities updated by the aggregation as needing a refresh. This lets you single out only updated identities for a refresh when the Refresh Identity Cube task is run. The default behavior of aggregation task is to set this flag; if you don't want an aggregation task to flag identities that need a refresh, you can turn this option off.

During the aggregation task, IdentityIQ marks the identities that have changed by setting the attribute needsRefresh to true on the changed identities as they are updated. This is a default operation performed in all aggregation tasks, although it can be turned off with an option on each of the aggregation tasks if desired.

This needsRefresh flag can then be used by the Identity Refresh tasks to target only those identities with accounts that were modified in a recent aggregation. The refresh tasks can then reset that flag to false when they are done with the identities so that subsequent aggregations can set the flag anew, and subsequent refresh cycles will again only pick up changed identities.

If you want to use this delta identity refresh feature, you should carefully consider which attributes you choose to aggregate from your applications into IdentityIQ. Aggregating attributes such as last login date, for example, would likely cause IdentityIQ to reflect changes to identities more frequently than choosing to aggregate only more static data fields, and would therefore flag more identities for delta refresh.

Note that aggregation is the only process which automatically sets this needsRefresh flag on identities. If other processes (such as Lifecycle Manager requests) make attribute or account changes to an identity which would affect identity refresh functionality, a full refresh that does not rely on this flag would be required to process those other identities' changes. Alternatively, the Lifecycle Manager workflows also include an optional targeted identity refresh step which could refresh the single changed identity immediately, and could be configured either to clear or not clear that identity's current needsRefresh flag value at that time.

To set up an aggregation task that will mark identities that have changed:

  1. Click Setup > Tasks.

  2. Choose the aggregation task to edit.

  3. Uncheck the Disable marking the identity as needing a refresh option.

  4. Save the task.

Refreshing Only Identities Marked As Changed

In the Refresh Identity Cube task (or other refresh tasks), select the option to Refresh only identities marked as needing refresh during aggregation. It is important to note that this operation is disabled by default; that is, default behavior for Identity Refresh tasks is to ignore the needsRefresh flag that was set by the aggregation task. If you want to use the delta identity refresh feature, you have to explicitly set this option in your refresh task(s):

  1. Click Setup > Tasks.

  2. Choose the refresh task to edit.

  3. Check the Refresh only identities marked as needing refresh during aggregation option.

  4. Save the task.

When the refresh task runs, it resets the needsRefresh flag to false for every identity it processes. This way, IdentityIQ knows that the identity has been refreshed already and so will not refresh it again until the next aggregation. However, you can change this behavior if you want. Depending on how you run refresh tasks, you may or may not want to reset this flag.

For example, if you aggregate and refresh infrequently, it can be a good practice to have the refresh task clear the needsRefresh tag, to avoid needlessly repeating refreshes on an identity that has just been refreshed. However, if you segment the refresh task to, for example, split out the refreshing of entitlements, attributes, and policies, you would not want to clear the needsRefresh tag. Leaving the needsRefresh tag in place as you iterate through all the refresh segments lets you avoid a situation where an identity is updated only for one segment of the full refresh process, rather than all segments that might apply.

To configure the refresh task so that it does not clear the needsRefresh flag from an Identity Cube when it runs:

  1. Click Setup > Tasks.

  2. Choose the refresh task to edit.

  3. Check the Do not reset the needing refresh marker after refresh option.

  4. Save the task.

Best Practices for Delta Identity Refresh

Delta identity refresh offers flexibility for managing the important identity refresh functions more efficiently. Here are some recommended best practices for using this functionality.

Delta identity refresh can be a helpful performance boost for IdentityIQ, as it streamlines time-intensive identity refresh processes. It is still strongly recommended, however, that customers regularly run a full refresh to ensure that all identities get updated and processed following other changes which may occur outside the data flows which would trigger identities to be marked as needing refresh. For example, Lifecycle Manager request workflows may or may not refresh identities after entitlement and role assignments, depending on options configured in the workflows, and they do not set the needsRefresh flag by default; a full refresh on a periodic basis would catch those identities and update them as needed.

Different customers run refreshes on different schedules, often depending on the volume of ongoing changes they anticipate or experience in their environment. For example, customers who run an hourly delta refresh should plan to run a daily full refresh, while customers with lower change volumes may run daily delta refreshes with weekly full refreshes.

It is common for installations to define workflow triggers (IdentityTrigger objects) which are set to run based on the calendar date when the trigger executes. For example, a "pre-offboarding" workflow could be configured to run the day before a user's termination date or a "pre-onboarding" workflow could run a day or a week before the user's start date, with those dates being specified through attributes in an HR or authoritative system feed. The Identity Refresh task that launches those workflow triggers (through its "Process Events" option) should be configured as a full refresh which processes all identities, not a delta refresh handling only identities that need refreshing; the Identity's attributes or accounts might not change in any aggregation run on the day these events need to execute, so this workflow needs to evaluate all identities, not just recently changed ones.

Logical applications in IdentityIQ are applications which are defined based on accounts, entitlements, and / or permissions granted through one or more other applications. For example, Application X, for which access is controlled by membership in the AppX group in LDAP, could be configured as a logical application in IdentityIQ.

Logical application "aggregation" is actually performed by processing information already stored in IdentityIQ for other applications (for example, to identify Application X accounts and entitlements, IdentityIQ looks at the LDAP application accounts with the AppX group), rather than by reading data from an external source like traditional application aggregations do. Standard applications, with a data source external to IdentityIQ, support an optimized aggregation process in which IdentityIQ skips the bulk of processing on any records it finds to be unchanged as it reads the source. Because the data source for logical applications is existing identity data which must be dynamically examined and assessed for logical accounts based on the logical application definition, there is no built-in option for optimizing aggregation from logical applications.

Delta identity refresh offers functionality which can be used to approximate an "optimized refresh," so it is possible to use this feature to manage a more optimized aggregation of logical applications. These are the steps to implement that option:

  1. Run an optimized aggregation from the tier applications in the logical application definition. This aggregation will then mark only the changed identities from those applications with the needsRefresh flag. For best results, this should be run at a time when there are no identities previous marked as needing refresh, that is, a full or delta refresh process has reset the needsRefresh flag on all marked identities prior to this aggregation.)

  2. Run a delta identity refresh task to recalculate the logical application accounts. Since it will only look at the identities whose accounts changed in the optimized aggregation, it will only recalculate the logical accounts for those identities. This refresh should have these options set:

    • Refresh only identities marked as needing refresh during aggregation – enable this option to have the task process only identities which were marked as changed in the optimized aggregations in step 1.

    • Refresh logical application links – enable this option to use this refresh for logical application account calculation.

    • Edit the task's definition in the Debug pages to add a compositeApplicationskey, set to a comma-separated list of logical application names which this refresh should process; this can be a single logical application at a time or a limited list so the task only processes the logical application(s) which are based on the tier applications just aggregated