Skip to content

Configuring and Scheduling the SharePoint Crawl

To configure the crawl:

  1. Go to Admin > Applications.
  2. Scroll through the list or use the filter to find the application.
  3. Select the Edit icon on the application row.

  4. Select Next until you reach the Crawler settings page.

    Note

    The entry fields vary by application type.

You can now schedule a task.

Scheduling a Task

To create a schedule:

  1. Select Create a Schedule.
  2. The system will provide a Schedule Task Name in the format {appName} - {type} Scheduler. Choose to keep or override this suggestion.

  3. Select a scheduling frequency from the dropdown list.

    Schedule Frequency Options
    • Run After - Create dependency of tasks. The task starts running only upon successful completion of the first task.
    • Hourly - Set the start time.
    • Daily - Set the start date and time.
    • Weekly - Set the day(s) of the week on which to run.
    • Monthly - Set the day of the month on which to run a task.
    • Quarterly - Set a monthly schedule with an interval of 3 months.
    • Half Yearly - Set a monthly schedule with an interval of 6 months.
    • Yearly - Set a monthly schedule with an interval of 12 months.

  4. Fill the Date and Time field with scheduling times. These fields differ depending upon the scheduling frequency selected.

  5. Select the Active checkbox to activate the schedule.
  6. Select Next.

Configuring and Scheduling the Crawler

To set or edit the Crawler configuration and scheduling:

  1. Go to Admin > Applications.
  2. Scroll through the list or use the filter to find the application.
  3. Select the Edit icon on the application row.

  4. Select Next until you reach the Crawler & Permissions Collection settings page.

    Note

    The entry fields vary by application type.

  5. In the Calculate Resource Size field, determine when, or at what frequency, Data Access Security calculates the resources' size:

    • Never
    • Always
    • Second crawl and on (default)

  6. Schedule a task.

  7. Set the Crawl Scope by:

Including and Excluding Paths by List

To set the paths to include or exclude in the crawl process for an application:

  1. Go to Admin > Applications.
  2. Scroll through the list or use the filter to find the application.
  3. Select the Edit icon on the application row.

  4. Select Next until you reach the Crawler & Permissions Collection settings page.

    Note

    The entry fields vary by application type.

  5. Scroll down to the Crawl configuration settings.

  6. Select Advanced Crawl Scope Configuration to open the scope configuration panel.
  7. Select Include / Exclude Resources to open the input fields.
  8. To add a resource to a list, enter the full path to include or exclude in the top field and select + to add it to the list.
  9. To remove a resource from a list, find the resource from the list, and select the x icon on the resource row.

Note

When creating exclusion lists, excludes take precedence over includes.

Excluding Paths by Regex

To set filters of paths to exclude in the crawl process for an application using regex:

  1. Go to Admin > Applications.
  2. Scroll through the list or use the filter to find the application.
  3. Select the Edit icon on the application row.

  4. Select Next until you reach the Crawler & Permissions Collection settings page.

    Note

    The entry fields vary by application type.

  5. Select Exclude Paths by Regex to open the configuration panel.

  6. Enter the paths to exclude by regex. Since the system does not collect Business Resources that match this regex, it also does not analyze them for permissions.

Crawler Regex Exclusion Example

The following are examples of crawler Regex exclusions:

Exclude all resources which start with one or more resource names:

Example: Starting with https://www.mysharepoint.com/resourceName

Regex: https:\/\/www.mysharepoint.com\/resourceName$

Example: Starting with https://www.mysharepoint.com\resourceName or //www.mysharepoint.com/OtherResourceName

Regex: https:\/\/www.mysharepoint.com\/(resourceName|OtherResourceName)$

Example: SharePoint resources starting with https://www.mysharepoint.com/sites/mySiteCollection

Regex: https:\/\/www.mysharepoint.com\/sites\/mySiteCollection$

Example: SharePoint resources starting with http://www.mysharepoint.com/sites/mySiteCollection or http://www.mysharepoint.com/other site/Different Site

Regex: https:\/\/www.mysharepoint.com\/(sites\/mySiteCollection|other_site\/Different_Site)$

Include ONLY resources which start with one or more resources names:

Example: Starting with https://www.mysharepoint.com/resourceName

Regex: ^(?!https:\/\/www.mysharepoint.com\/resourceName($|\/.)).

Example: Starting with https://www.mysharepoint.com/resourceName or https://www.mysharepoint.com/OtherResourceName

Regex: ^(?!https:\/\/www.mysharepoint.com\/(resourceName|OtherResourceName)($|\/.)).

Example: SharePoint resources starting with https://www.mysharepoint.com/sites/mySiteCollection

Regex: ^(?!https:\/\/www.mysharepoint.com\/sites\/mySiteCollection($|\/.)).

Example: SharePoint resources starting with https://www.mysharepoint.com/sites/mySiteCollection or https://www.mysharepoint.com/other site/Different_Site

Regex: ^(?!https:\/\/www.mysharepoint.com\/(sites\/mySiteCollection|other_ site\/Different_Site)($|\/.)).

Excluding Top-Level Resources

Use the top-level exclusion screen to select top-level roots to exclude from the crawl. This setting is done per application.

To exclude top-level resources from the crawl process:

  1. Go to Admin > Applications.
  2. Find the application to configure and select the dropdown list menu on the application line. Select Exclude Top Level Resources to open the configuration panel.
  3. Select the Run Task button to trigger a task that runs a short detection scan to detect the current top-level resources. If the top-level resource list has changed in the application while you are on this screen, select the Run Task button to retrieve the updated structure.
  4. Once triggered, you can view the task status in Settings > Task Management > Tasks, depending on your access to the task page.
  5. When the task has completed, select Refresh to update the page with the list of top-level resources.

  6. Select the top-level resource list and choose top-level resources to exclude.

    Note

    If all resources are selected and you wish for them to be deselected, select Deselect All. You can also select individual resources.

  7. Select Save to save the change.

  8. To refresh the list of top-level resources, run the task again. Running the task will not clear the list of top-level resources to exclude.

Documentation Feedback

Feedback is provided as an informational resource only and does not form part of SailPoint’s official product documentation. SailPoint does not warrant or make any guarantees about the feedback (including without limitation as to its accuracy, relevance, or reliability). All feedback is subject to the terms set forth at https://developer.sailpoint.com/discuss/tos.