# Pagination Web Services supports generic paging for Account and Entitlement Aggregations. > [!NOTE] > > **Notes** > > - If there are multiple Account or Entitlement Aggregation operations (endpoints) configured, paging is supported only for the first operation instance of Account or Entitlement Aggregation. > - By default, the page size is 50 and page offset is 0. > - The Web Services source correctly evaluates the `)` character within a string in the paging configuration. The following are methods to embed paging in the Web Services connector: - [Before and After Operation Rules Configuration](<#before-and-after-operation-rules-configuration>) - [Configure Paging](<#configure-paging>) ## Before and After Operation Rules Configuration To embed pagination in the Web Services connector, manual processing is required in the Before and After Operation Rules. Refer to [Web Services Before/After Operation Rule]() for more information on the rules. The Web Services connector relies on temporary information stored in the application object in the `transientValues` map. ### Paging with searchAfter Parameter The Web Services connector supports paging with the Before Rule and After Rule using the `searchAfter` array parameter. This parameter is used in the Before Rule to modify the request body for paginated search results. - `searchAfter` array - An array used in the Before Rule to modify the request body for paginated search results. - `hasMore` - A Boolean parameter that explicitly indicates whether more records are available for aggregation. #### Web Services Before Rule for Paging The Before Rule alters the URL and request parameters if the value of the `hasMore` parameter is set to `true` and the request to aggregate additional accounts is triggered. If the `hasMore` parameter is not set or it is set to `false`, the pagination request terminates, except for the initial request. The administrator must write the Before Rule for account and group aggregation properly. To support the `searchAfter` array parameter in pagination, refer to the following Before Rule sample for the account aggregation request in Perform Search: ```java import connector.common.JsonUtil; import java.util.Map; //The hasMore parameter must be set to TRUE by an earlier pagination process' AfterRule setting. //This parameter setting is mandatory as it acts as the deciding factor for terminating the pagination request. //If the hasMore parameter is not set, or if it is set to FALSE, the pagination request terminates for every //aggregation process except the initial process. Map transientValues = application.getAttributeValue("transientValues"); //The connector checks if it is the initial request or not. If it is, //then it executes without the searchAfter parameter. //If it is not the initial request, then the connector checks the search //after value(s) updated in the transientValues map in the After Operation //rule. It updates the searchAfter parameter with value in the request body. if(transientValues != null ) { log.debug("Before Rule: transientValues-->"+JsonUtil.render(transientValues)); Map body = requestEndPoint.getBody(); String jsonBody = (String) body.get("jsonBody"); Map jsonMap = JsonUtil.toMap(jsonBody); jsonMap.put("searchAfter", transientValues.get("searchAfterArray")); String finalBody = JsonUtil.render(jsonMap); body.put("jsonBody", finalBody); log.debug("Before Rule: Request Body-->"+jsonMap); requestEndPoint.setBody(body); } return requestEndPoint; ``` The Web Services Before Rule checks for transient values. If transient values are present, then it updates the request body with the `searchAfter` parameter. #### Web Services After Rule for Paging The After Rule determines whether the managed system has more records that can be aggregated and added as entries in the `transientValues` map. It also sets the `hasMore` key with a value of `true` or `false` depending upon the number of records aggregated. The After Rule stores the `searchAfter` array and `hasMore` parameter values in the `transientValues` map within the application object. This map stores the necessary information used by the Before Rule to manipulate the next API call. > [!IMPORTANT] > > Ensure that the parameter indicating whether the managed system contains more records is stored by the key named `hasMore`. This field is mandatory as it is the deciding factor for aborting the pagination requests. In the following example, the `searchAfter` entry is added into the `transientValues` map. The last object of the `processedResponseObject` list is used to get the `searchAfter` parameter value. This value determines how sorting works. The administrator must write the After Rule for account and group aggregation properly. Refer to the following sample After Rule for account aggregation requests in Perform Search: > [!NOTE] > > The provided example works correctly as long as no updates are needed in the resource objects. It may trigger a warning message in the logs, however it functions as intended. ```java import java.util.Map; //The expected number of objects from the pagination result. Integer expectedRecordsCount = 250; //The connector then checks if processedResponseObject is null or empty. //If it's null or empty then there's no need to process the request. if (processedResponseObject!= null && !processedResponseObject.isEmpty()) { //The connector fetches the last object to get the updated value of the searchAfter parameter. Map lastRecord = processedResponseObject.get(processedResponseObject.size()-1); //Attribute value on which sorting works, this attribute value will be used in the searchAfter parameter. String id = (String)lastRecord.get("id"); String[] searchAfterArray= new String[]{id}; log.debug("AFTER RULE: searchAfterArray ==> " + id); Integer fetchedRecordsCount = 0; if(null != processedResponseObject) { fetchedRecordsCount = ((List) processedResponseObject).size(); } //If the fetched records are equal to the expected number of records, //the hasMore value will be set to TRUE and pagination will continue. boolean hasMore = (fetchedRecordsCount != 0 && null != expectedRecordsCount && fetchedRecordsCount.equals(expectedRecordsCount)); log.debug("AFTER RULE: Has More? ==> " + hasMore); Map transientValues = application.getAttributeValue("transientValues"); if(transientValues == null) { transientValues = new HashMap(); application.setAttribute("transientValues", transientValues); } transientValues.put("hasMore", hasMore); if(hasMore){ transientValues.put("searchAfterArray", searchAfterArray); } } ``` ### Paging for Dropbox The administrator must write the Before Rule and After Rule for Dropbox account and group aggregation as follows. #### Dropbox Before Rule See the sample Before Rule for the account aggregation request in the Web Services connector for Dropbox using V2 in `examplerules.xml` file by name `Example WSBeforeRl DropboxPaging` as follows: ```java import sailpoint.tools.Util; Map obj = (Map) application.getAttributeValue("transientValues"); System.out.println("BEFORE RULE: Transient Values ==> " + obj); if(null != obj) { String offset = obj.get("offset"); System.out.println("BEFORE RULE: offset value ==> " + offset); String urlString = (String) requestEndPoint.getFullUrl(); if(Util.isNotNullOrEmpty(offset)) { System.out.println("BEFORE RULE: requestEndpoint ==> " + requestEndPoint); System.out.println("BEFORE RULE: URL ==> " + urlString); URL tempUrl = new URL(urlString); String queryString = tempUrl.getQuery(); System.out.println("BEFORE RULE: Query String ==> " + queryString); if(Util.isNotNullOrEmpty(queryString)) { StringBuffer queryParams = new StringBuffer(); String[] params = tempUrl.getQuery().split("&"); for (String param : params) { if(queryParams.length() > 0) queryParams.append("&"); if(param.startsWith("sysparm_offset=")) { queryParams.append("sysparm_offset="); queryParams.append(offset); } else { queryParams.append(param); } } urlString = urlString.replace(tempUrl.getQuery(), queryParams.toString()); } } System.out.println("BEFORE RULE: Updated Query String ==> " + urlString); requestEndPoint.setFullUrl(urlString); } System.out.println("BEFORE RULE: requestEndpoint Updated ==> " + requestEndPoint); return requestEndPoint; ``` For Dropbox V2, the Web Services After Rule stores the cursor returned from the previous team membership listing API in the `transientValues` map in the application. The URL is modified to redirect to the paging API and the cursor is sent as a part of the form data. Ensure that the `hasMore` flag is set by the earlier requests of the After Rule. #### Dropbox After Rule Configuring paging for Dropbox has specific instructions. Refer to the sample After Rule for the account aggregation request in the Web Services connector for Dropbox using V2 in `examplerules.xml` by name `Example WSAfterRl DropboxPaging` as follows: ```java Integer fetchedRecordsCount = 0; if(null != processedResponseObject) { fetchedRecordsCount = ((List) processedResponseObject).size(); } Integer expectedCount = null; Integer offset = null; URL url = new URL(requestEndPoint.getFullUrl()); System.out.println("AFTER RULE: Original Url ==> " + url); String[] params = url.getQuery().split("&"); for (String param : params) { String name = param.split("=")[0]; String value = param.split("=")[1]; switch(name) { case "sysparm_limit": expectedCount = Integer.parseInt(value); break; case "sysparm_offset": offset = Integer.parseInt(value); break; default: } } System.out.println("AFTER RULE: Fetch Count ==> " + fetchedRecordsCount); System.out.println("AFTER RULE: Limit Count ==> " + expectedCount); System.out.println("AFTER RULE: Fetch Offset ==> " + offset); Boolean hasMore = (fetchedRecordsCount != 0 && null != expectedCount && fetchedRecordsCount.equals(expectedCount) && null != offset); System.out.println("AFTER RULE: Has More? ==> " + hasMore); Map transientValues = application.getAttributeValue("transientValues"); if(transientValues == null) { transientValues = new HashMap(); application.setAttribute("transientValues", transientValues); } transientValues.put("hasMore", hasMore); if (hasMore) { if(null != offset) { System.out.println("AFTER RULE: New Offset ==> " + (offset + expectedCount)); transientValues.put("offset", String.valueOf(offset + expectedCount)); } } ``` The Team membership response in Dropbox and Dropbox V2 contains the following elements: - `cursor` - An encrypted token that represents the next page to be aggregated, if any, and forms part of the subsequent API calls. - `has_more` - A Boolean value that explicitly indicates whether more records are available for fetching. The After Rule stores the `cursor` and `has_more` values from the response in the `transientValues` map in the application object. This map stores the necessary information used by the Before Rule to manipulate the next API call. > [!IMPORTANT] > > Ensure that the parameter indicating whether the managed system contains more records is stored by the key named `has_more`. This field is mandatory as it is the deciding factor for aborting the pagination requests. ## Configure Paging Paging can be configured in Account/Entitlement Aggregation endpoints using the **Paging** tab with one of the following methods: - Paging based on limit-offset - Paging based on response markers - Paging based on response header links > [!NOTE] > > The **Paging** tab becomes available when you configure certain HTTP operations, such as account and entitlement aggregation. Refer to [Configuring Paging as an HTTP Operation](<#configuring-paging-as-an-http-operation>) for more information on configuring paging as an HTTP operation. ### Configuration Considerations Consider the following while configuring paging in the **Paging** tab: - Every paging configuration step must start on a new line. - SailPoint recommends providing a space after every operator, condition, or placeholder for correct evaluation of the paging expression. - The paging mechanism follows the placeholder notation for resolution of attribute values: ```text $response.attribute_key$ ``` Any attribute which follows the placeholder notation is resolved or assigned a value depending upon the operator being used. - Intermediate values can also be stored between page requests by using the placeholder notation. To achieve this, any attribute key which does not match any of the predefined keywords can be used. For more information, see the example in [Paging Based on Limit-Offset](<#paging-based-on-limit-offset>) where `$sysparm_offset$` is updated and used between page requests. - For complex expressions or conditions, multiple conditions can be grouped together using parentheses. For example: ```text TERMINATE_IF ($someattribute$ == TRUE) && ($otherattribute$ == NULL) ``` > [!IMPORTANT] > > For more information and implementation of paging, contact SailPoint Services. ### Paging Keywords The paging mechanism has the following predefined set of keywords you can use: | Keyword | Description | | --- | --- | | `application` | Represents the application. `baseUrl` - Base URL configured in the source. | | `endpoint` | Represents endpoint configuration. `relativeURL` - Relative URL of the endpoint. `fullUrl` - Full URL contained within the endpoint. | | `limit` | Indicates page limit configured in the endpoint configuration. | | `offset` | Indicates page offset configured in the endpoint configuration. | | `request` | Represents request body. | | `requestHeaders` | Represents request header. | | `response` | Represents response body object. | | `responseHeaders` | Represents response header. | | `TERMINATE_IF` | Indicates termination condition; multiple conditions can exist. `NO_RECORDS` - Indicates no records retrieved; to be used in conjunction with `TERMINATE_IF`, evaluates to TRUE/FALSE. `RECORDS_COUNT` - Indicates number of records retrieved. | | `NULL` | Indicates null or empty object. | | `REMOVE` | Indicates to remove an attribute. | > [!IMPORTANT] > > Avoid using arithmetic operations in the `TERMINATE_IF` statement. Arithmetic statements should be performed before checking for termination conditions. ### Supported Operators | Type | Operations | | --- | --- | | Regular operations | `+`, `-`, `*`, `/`, `=`, `&&`, and `\|\|` | | Conditional operations | `<`, `>`, `<=`, `>=`, `==`, and `!=` | ### Paging Based on Request Body Configuration Configure paging using request body parameter configuration. For example, an initial aggregation URL for an example managed system is as follows: ```text https://test.com/GetUserList ``` The example URL has the following parameters: - **baseUrl** - `https://test.com` - **relativeURL** - `/GetUserList` The request body is as follows: ```text Body: { "Limit" : 50, "offset": 1 } ``` Use the following configuration steps on the **Paging** tab based on the request body: ```text $limit$ = 50 TERMINATE_IF $RECORDS_COUNT$ < $limit$ $offset$ = $offset$ + 1 $endpoint.fullUrl$ = $application.baseUrl$ + "/GetUserList" $request.PageIndex$ = $offset$ ``` ### Paging Based on Limit-Offset You can limit pagination based on the `$offset$` and `$limit$` placeholders. - `$offset$` - Initial page offset - `$limit$` - Page size For example, in the following configuration, the maximum number of users to fetch per page is 100, and the starting offset of the API is 0. The index in the subsequent page is 101. 1. In **General Information \> Context URL**, enter the following: ```text /v2/users?pageSize=$limit$&index=$offset$ ``` 2. Go to the **Paging** tab and set the following: - **Page Size** - `100` - **Initial Page Offset** - `0` 3. Configure the following in **Paging Steps**: ```text TERMINATE_IF $RECORDS_COUNT$ < $limit$ $offset$ = $offset$ + $limit$ $endpoint.fullUrl$ = $application.baseUrl$ + "/v2/users?pageSize=" + $limit$ + "&index=" + $offset$ ``` #### Managed System Specific Configurations This section describes paging based on limit-offset for ServiceNow and Workday managed systems. ##### For ServiceNow For example, the initial aggregation URL for a ServiceNow managed system is as follows: ```text https://XYZ.service-now.com/api/now/v1/table/sys_user?sysparm_limit=100&sysparm_fields=sys_id ``` The above URL includes the following parameters: - **baseUrl** - `https://XYZ.service-now.com` - **relativeURL** - `api/now/v1/table/sys_user?sysparm_limit=100&sysparm_fields=sys_id` To configure paging based on limit-offset: 1. Use the following for a ServiceNow managed system: ```text TERMINATE_IF $RECORDS_COUNT$ < $sysparm_limit$ $sysparm_offset$ = $sysparm_offset$ + $sysparm_limit$ $endpoint.fullUrl$ = $application.baseUrl$ + "/api/now/v1/table/sys_user?sysparm_fields=sys_id&sysparm_limit=100&sysparm_offset=" + $sysparm_offset$ ``` In this example, the maximum record count is set using `sysparm_limit` and `RECORDS_COUNT`. `RECORDS_COUNT` represents the number of records fetched from the response and `sysparm_limit` can be changed as required. Based on this step, the next page URL is as mentioned in the following steps. 2. The second page URL is based on the paging steps configuration: ```text https://XYZ.com/api/now/v1/table/sys_user?sysparm_limit=100&sysparm_fields=sys_id&sysparm_offset=100 ``` 3. The next page URL is based on the paging steps configuration: ```text https://XYZ.com/api/now/v1/table/sys_user?sysparm_limit=100&sysparm_fields=sys_id&sysparm_offset=200 ``` The paging operation is terminated when `RECORDS_COUNT` is less than `sysparm_limit`, as follows: ```text $RECORDS_COUNT$ < $sysparm_limit$ ``` ##### For Workday For example, the initial aggregation URL for a Workday managed system is as follows: ```text https://XYZ.workday.com/ccx/service/sailpoint_pt1/Human_Resources/v24.1 ``` The above URL includes the following parameters: - **baseUrl** - `https://XYZ.workday.com` - **relativeURL** - `/ccx/service/sailpoint_pt1/Human_Resources/v24.1` The following are the configuration steps in the paging tab based on limit-offset: 1. Use the following for a Workday managed system: ```text TERMINATE_IF $response.wd:Response_Results.wd:Page$ > $response.wd:Response_Results.wd:Total_Pages$ $offset$ = $response.wd:Response_Results.wd:Page$ + 1 $request.bsvc:Response_Filter.bsvc:Page.text()[1]$ = $offset$ ``` In the above example, the number of pages is verified with the response from Workday and compared with the total number of pages. If the current page number is less than the total pages, then the request body (SOAP Body) is updated with the new page by incrementing it. The following is an example of an initial payload request with Initial Page Size (the initial page is the beginning index) as 1 and Page Size (number of records per page) as 10: ```xml .... 1 10 .... ``` 2. Based on the previous paging step, the next payload is as follows. The second aggregation payload increments the offset (page number): ```xml .... 2 10 .... ``` ### Paging Based on Response Markers This section describes paging based on a marker value in the response. Based on the marker value in the response, an aggregation process is terminated or continued. The following table lists examples of response marker based paging for Dropbox, Salesforce, and SuccessFactors. > [!NOTE] > > Other managed systems may use different response marker formats. In some configurations, it may be better to use the Before or After Operation Rule to configure paging. Refer to [Web Services Before/After Operation Rule]() for more information. #### Dropbox ```text TERMINATE_IF $response.has_more$ == FALSE $endpoint.fullUrl$ = $application.baseUrl$ + $endpoint.relativeUrl$ + "/continue" $request.cursor$ = $response.cursor$ REMOVE $request.limit$ ``` #### Salesforce ```text TERMINATE_IF $response.ns:result.ns:done$ != FALSE $request.soapenv:Body$ = "" + $response.ns:result.ns:queryLocator$ + "" ``` #### SuccessFactors ```text TERMINATE_IF $response.ns:result.ns:hasMore$ != TRUE $request.soapenv:Body$ = "" + $response.ns:result.ns:querySessionId$ + "" ``` ### Paging Based on Response Header Links This section describes paging based on response header links. The following table lists examples of response header links-based paging for Basware, ServiceNow, and Okta managed systems. > [!NOTE] > > Other managed systems may use different response header link formats. In some configurations, it may be better to use the Before or After Operation Rule to configure paging. Refer to [Web Services Before/After Operation Rule]() for more information. #### Basware ```text TERMINATE_IF $responseHeaders.x-amz-meta-continuationtoken$ == NULL $requestHeaders.x-amz-meta-continuationtoken$ = $responseHeaders.x-amz-meta-continuationtoken$ ``` #### ServiceNow ```text TERMINATE_IF $responseHeaders.link.next$ == NULL $endpoint.fullUrl$ = $responseHeaders.link.next$ ``` #### Okta ```text TERMINATE_IF $responseHeaders.link.next$ == NULL $endpoint.fullUrl$ = $responseHeaders.link.next$ ``` ## Configuring Paging as an HTTP Operation Configure paging as an HTTP operation. The **Paging** option appears for aggregation operations. Refer to [Pagination](<#pagination>) for more information on the paging feature. > [!NOTE] > > **Screenshot** > > [![Paging]()]() 1. Set the **Initial Page Offset**, which sets the page offset configured in the endpoint configuration. 2. Set the **Page Size**, which determines how many records the connector displays on each page. 3. In **Paging Steps**, enter configuration steps as required for limit-offset, response markers, or response header links paging. 4. Select **Save**.