Validating Data Extract Results

If your implementation uses the embedded ActiveMQ message broker, you can verify extracted messages in the ACTIVEMQ_MSGS table. There is not a way to monitor temporary queues.

If your implementation uses external ActiveMQ, you can see the following happen as a Data Extract task runs:

  • OBJECT_SELECTOR messages are written in the inbound queue.

    • This is the list of object IDs / names to extract.

    • The partitions / threads will read messages from this queue then process and write to the destination queue.

  • EXTRACTED_OBJECT messages are written to the destination queue with the name specified in your Extract YAMLConfig.

To verify the content of EXTRACTED_OBJECT messages, you can do the following:

  1. Run Task with Transform / Extract config to extract all objects of all types (generate transform config for all objects) and verify messages.

  2. Run Task with Transform / Extract config to extract all objects for single object type.

  3. Run Task with Transform / Extract config to extract all objects for multiple object types.

  4. Run Task with Transform / Extract config with filterString to restrict which objects of a particular type to extract.

Verify that the number of messages match:

  • Using Embedded Message Broker: In the task result, verify that the number of message in the table for destination queue (see Debug > ActiveMQ Monitoring, scroll down to Destinations) matches the number of Extracted Objects/Extracted Objects Dispatched (see Task Results).

  • Using External Broker: Verify that the number of messages in the temporary queue match the number in the destination queue. Additionally, the task result should show the same number for Extracted Objects as for Extracted Objects Dispatched.

Spot check messages:

  1. Generate the default Transform YAMLConfig for identity. Add filterString to Extract YAMLConfig to extract single identity. Run the task using that config and view the dispatched message content, verify data looks correct.

  2. Remove some properties or attributes from the Transform YAMLConfig and repeat. View dispatched message content, verify that it's correct.

  3. Repeat with some other tasks.