Paging with searchAfter Parameter

The Web Services connector supports paging with the Before Rule and After Rule using the searchAfter array parameter. This parameter is used in the Before Rule to modify the request body for paginated search results.

  • searchAfter Array – An array which will be used in the Before Rule to modify the request body for paginated search results.

  • hasMore – A Boolean parameter which explicitly indicates whether more records are available for aggregation.

Web Service Before Rule for Paging

The Before Rule alters the URL/request parameters if the value of the hasMore parameter is set to true and the request to aggregates additional accounts is triggered. If the hasMore parameter is not set or it is set to false, the pagination request terminates, except for the initial request.

The administrator must write the Before Rule for account and group aggregation properly.

To support searchAfter array parameter in pagination, see the Before Rule sample below for account aggregation request in Perform Search in examplerules.xml by name WebService SearchAfter Paging example BeforeOperationRule as follows:

Copy 
import connector.common.JsonUtil;
import java.util.Map;

//The hasMore parameter must be set to TRUE by an earlier pagination process' AfterRule setting.
//This parameter setting is mandatory as it acts as the deciding factor for terminating the pagination request.
//If the hasMore parameter is not set, or if it is set to FALSE, the pagination request terminates for every 
//aggregation process except the initial process.

Map transientValues = application.getAttributeValue("transientValues");    

//The connector checks if it is the initial request or not. If it is,
//then it executes without the searchAfter parameter.
//If it is not the initial request, then the connector checks the search 
//after value(s) updated in the transientValues map in the After Operation
//rule. It updates the searchAfter parameter with value in the request body.

if(transientValues != null ) {                                                                                    
    log.debug("Before Rule: transientValues-->"+JsonUtil.render(transientValues));
    Map body = requestEndPoint.getBody();
    String jsonBody = (String) body.get("jsonBody");
    Map jsonMap = JsonUtil.toMap(jsonBody);
    jsonMap.put("searchAfter", transientValues.get("searchAfterArray"));
    String finalBody = JsonUtil.render(jsonMap);
    body.put("jsonBody", finalBody);
    log.debug("Before Rule: Request Body-->"+jsonMap);
    requestEndPoint.setBody(body);
}
return requestEndPoint;

The Web Service Before Rule checks for transient values. If transient values are present, then it updates the request body with the searchAfter parameter.

Web Service After Rule for Paging

The After Rule determines whether the managed system has more records which can be aggregated and added as entries in the transientValues map. It also sets the hasMore key with a value as true or false depending upon the number of records aggregated.

The After Rule stores the searchAfter array and hasMore parameter values in the transientValues map within the Application object. This map stores the necessary information used by the Before Rule to manipulate the next API call.

Important
Ensure that the parameter indicating whether the managed system contains more records is stored by the key is named "hasMore". This field is mandatory as it is the deciding factor for aborting the pagination requests.

In the following example, the searchAfter entry will be added into the transientValues map. The last object of the processedResponseObject list will be used to get the searchAfter parameter value. This value determines how sorting works.

The administrator must write the After Rule for account and group aggregation properly.

See the sample After Rule for account aggregation requests in Perform Search in examplerules.xml by name WebService SearchAfter Paging example AfterOperationRule as follows:

Note
The provided example will work correctly as long as no updates are needed in the Resource Objects. It may trigger a warning message in the logs, however it will function as intended.

Copy 
import java.util.Map;

//The expected number of objects from the pagination result.

Integer expectedRecordsCount = 250;

//The connector then checks if processedResponseObject is null or empty.
//If it's null or empty then there's no need to process the request.

if (processedResponseObject!= null && !processedResponseObject.isEmpty()) {

//The connector fetches the last object to get the updated value of the searchAfter parameter.

    Map lastRecord = processedResponseObject.get(processedResponseObject.size()-1);
    
//Attribute value on which sorting works, this attribute value will be used in the searchAfter parameter.    
    
    String id = (String)lastRecord.get("id");
    String[] searchAfterArray= new String[]{id};
    log.debug("AFTER RULE: searchAfterArray ==> " + id);

    Integer fetchedRecordsCount = 0;
        if(null != processedResponseObject) {
            fetchedRecordsCount = ((List) processedResponseObject).size();
            }                                                                          
            
//If the fetched records are equal to the expected number of records,
//the hasMore value will be set to TRUE and pagination will continue.
            
            boolean hasMore = (fetchedRecordsCount != 0 && null != expectedRecordsCount && fetchedRecordsCount.equals(expectedRecordsCount));
            log.debug("AFTER RULE: Has More? ==> " + hasMore);
            Map transientValues = application.getAttributeValue("transientValues");
        if(transientValues == null) {
            transientValues = new HashMap();
            application.setAttribute("transientValues", transientValues);
            }
            transientValues.put("hasMore", hasMore);
        if(hasMore){
            transientValues.put("searchAfterArray", searchAfterArray);
    }
}