# Dataset Management

Dataset Management lets you review the object types a source collects and control how the source aggregates those objects into SailPoint. Use Dataset Management to view and manage resource attributes, datasets and their resources, enable dataset aggregation schedules, perform manual aggregation, and view aggregation history.

> [!NOTE]
>
> Datasets are also available in SailPoint Agentic Fabric. Contact your Customer Success team and refer to [SailPoint Agentic Fabric](<https://documentation.sailpoint.com/saas/help/agentic_fabric/index.html>) for more information.

For more information, refer to [Managing Datasets and Resources](<https://documentation.sailpoint.com/saas/help/sources/datasets.html>) for datasets, resources, owner correlation, and aggregation.

## Datasets

A *dataset* is a named group of one or more related resources that the source aggregates together.

When aggregation runs for a dataset, the source collects all resources in that dataset from the managed system in one aggregation job. Activate or deactivate aggregation at the dataset level, not for individual resources.

> [!NOTE]
>
> Each resource belongs to exactly one dataset for that source. The source defines which resources belong to each dataset. You can enable aggregation and review associated resources, but you cannot create or delete datasets or move resources between them.

### Dataset Aggregation

Configure aggregation on the **Dataset Aggregations** tab for each dataset. Configure each dataset separately. During aggregation, your owner correlation rules are automatically applied to link machine identities to their human owners. For more information, refer to [Configuring Dataset Aggregations](<https://documentation.sailpoint.com/saas/help/sources/datasets.html#configuring-dataset-aggregations>).

## Resources

A resource defines an object type from the managed system. It specifies the kind of object, the attributes the source collects, and how the source identifies and displays each object. Resources form the building blocks of the data model.

In the UI, each resource appears under **Dataset Management** \> **Resources** with its schema. A resource does not run aggregation by itself. The dataset that owns the resource collects it during aggregation.

For more information, refer to [Managing Resources](<https://documentation.sailpoint.com/saas/help/sources/datasets.html#managing-resources>).

### Owner Correlation

Configure owner correlation on each resource to link aggregated objects to human owners. If you do not configure owner correlation, or if the mapping produces no match, an owner is not assigned. For more information, refer to [Configuring Owner Correlation](<https://documentation.sailpoint.com/saas/help/sources/datasets.html#configuring-owner-correlation>).

## Supported Datasets

Go to **Dataset Management \> Datasets** on your Cluvio source to view the datasets the connector aggregates. Select a dataset to review its associated resources and configure aggregation.

> [!NOTE]
>
> Configure aggregation for each dataset separately. For more information, refer to [Dataset Aggregation](<https://documentation.sailpoint.com/saas/help/sources/datasets.html#configuring-dataset-aggregations>).

| Dataset name | Associated resources | Dataset ID | Aggregation schedule |
| --- | --- | --- | --- |
| **Datasource Agents** | Datasource Agent | `cluvio:datasource-agents` | Every 12 hours (`0 */12 * * *`) |
| **Datasources** | Datasource | `cluvio:datasources` | Every 12 hours (`0 */12 * * *`) |

For schema attribute definitions, refer to [Datasource Agent Attributes](<#datasource-agent-attributes>) and [Datasource Attributes](<#datasource-attributes>).

### Dataset Aggregations

Dataset aggregation collects resource objects from the managed system and loads them into SailPoint. Repeat the configuration for each dataset on the source.

### Configuring Dataset Aggregation

To configure dataset aggregation for a dataset:

1. Go to **Admin \> Connections \> Sources**, and open your source.
2. Go to **Dataset Management \> Datasets**.
3. Select the dataset you want to configure.
4. Select the **Dataset Aggregations** tab.

   On the **Dataset Aggregations** tab, configure how resource objects from the selected dataset are aggregated into SailPoint. The tab contains **Scheduled Aggregations**, **Manual Aggregation**, and **Latest Aggregation**.

#### Scheduled Aggregations

Scheduled aggregations run on a schedule to keep the dataset synchronized with the managed system.

1. On the **Dataset Aggregations** tab, select **Enable Schedule** to turn on scheduled aggregations.

When scheduled aggregations are enabled, SailPoint aggregates all resource objects in the dataset on the configured schedule.

#### Manual Aggregation

Manual aggregation runs once and collects all resource objects in the dataset.

1. On the **Dataset Aggregations** tab, select **Start Aggregation**.

SailPoint starts the aggregation immediately. When the run completes, results appear in the **Latest Aggregation** section.

#### Viewing Aggregation Results

After an aggregation completes, review its status in the **Latest Aggregation** section on the **Dataset Aggregations** tab.

The **Latest Aggregation** section shows:

- **Start Time** - when the aggregation started
- **Resource Objects Scanned** - the number of resource objects the aggregation processed
- **Status** - whether the aggregation succeeded or failed

Select **Details** to view additional information about the most recent run. To view prior runs, select **Aggregation History**.

## Supported Resources

For more information, refer to [Managing Resources](<https://documentation.sailpoint.com/saas/help/sources/datasets.html#managing-resources>).

> [!NOTE]
>
> When you enable dataset aggregation for a selected dataset, all resources in that dataset are collected at once. Activate or deactivate aggregation at the dataset level, not for individual resources.

Under **Dataset Management** \> **Resources**, the Cluvio source lists each object type that is aggregated. Select a resource to review its schema on the **Schema** tab.

| Resource name | Resource type | Resource ID |
| --- | --- | --- |
| **[Datasource Agent](<#datasource-agent-attributes>)** | `std:agent` | `cluvio:datasource-agent` |
| **[Datasource](<#datasource-attributes>)** | `std:credential` | `cluvio:datasource` |

### Owner Correlation

On the **Owner Correlation** tab of each resource, map the aggregated object to the identity that owns it. For more information, refer to [Owner Correlation](<https://documentation.sailpoint.com/saas/help/sources/datasets.html#configuring-owner-correlation>).

## Datasource Agent Attributes

The **Datasource Agent** resource is in the **Datasource Agents** dataset.

| Field | Value |
| --- | --- |
| Resource ID | `cluvio:datasource-agent` |
| Resource type | `std:agent` |
| Identity attribute | `ID` |
| Display attribute | `Name` |

The following table lists the schema attributes for the Datasource Agent resource:

| Attribute Name | Description |
| --- | --- |
| ID | ID of the datasource agent. |
| Agent ID | Agent ID of the datasource agent. |
| Name | Name of the datasource agent. |
| Version | Last known version of the datasource agent. |
| Address | Last known IP address of the datasource agent. |
| Connected | Whether the datasource agent is connected. |
| Active Queries | Approximate number of running queries on the datasource agent. |
| Enabled | Whether the datasource agent is enabled. Only enabled agents can connect to Cluvio. |
| Datasources | IDs of datasources that use this datasource agent. |

## Datasource Attributes

The **Datasource** resource is in the **Datasources** dataset.

| Field | Value |
| --- | --- |
| Resource ID | `cluvio:datasource` |
| Resource type | `std:credential` |
| Identity attribute | `ID` |
| Display attribute | `Name` |

> [!NOTE]
>
> The connector does not aggregate **Password** or **BigQuery Service Key**.

The following table lists the schema attributes for the Datasource resource:

| Attribute Name | Description |
| --- | --- |
| ID | ID of the datasource. |
| Name | Name of the datasource. |
| Database Type | Database type of the datasource. |
| Username | Database username used when connecting. For BigQuery this is the project ID. |
| Database | Database name to connect to. Not applicable to BigQuery. |
| Schema Names | Schema names returned for the datasource. |
| Monitoring Enabled | Whether monitoring is enabled for the datasource. |
| Host | Database hostname to connect to. |
| Port | Database port to connect to. |
| Connection Mode | Connection mode of the datasource. Cluvio returns direct, ssh\_tunnel, or agent. |
| Require SSL | Whether the datasource connection requires SSL. |
| Verify SSL | Whether the datasource connection verifies SSL. |
| Data Time Zone | Data time zone used for the datasource. |
| Enabled | Whether the datasource is enabled. |
| Created At | Date and time when the datasource was created. |
| Updated At | Date and time when the datasource was last updated. |
| Schema Updated At | Date and time when the datasource schema was last updated. |
| Total Row Count | Total row count reported for the datasource. |
| Total Row Count Exact | Whether the reported total row count is exact. |
| Default Schema Name | Default schema name of the datasource. |
| Server Version | Database server version reported for the datasource. |
| Max Concurrent Executions | Maximum number of concurrent executions for the datasource. |
| Update Schema Nightly | Whether nightly schema updates are performed for the Cluvio almanac. |
| Update Schema Row Counts | Whether schema updates include row counts. |
| Included Schemas | Schemas included in the Cluvio almanac. |
| Included Databases | Databases, catalogs, or projects included in the Cluvio almanac. |
| Is Sample | Whether the datasource is a sample datasource. |