Working with Datasets

This guide explains how to work with datasets in Pretectum using the API. You will learn how to retrieve datasets, understand their properties, and use them effectively for organizing and searching your data.

Overview

Datasets are collections of data objects (records) that share the same schema structure. They represent the actual data stored in your Pretectum master data repository and provide a way to organize records into logical groups. The hierarchy in Pretectum is:

Business Area (e.g., Customer)
  └── Schema (e.g., Individual Customer)
       └── Dataset (e.g., US Customers, European Customers)
            └── Data Objects (actual customer records)

Why Datasets Matter

Datasets help you:

Organize Data: Group records by region, source, time period, or any logical category
Filter Searches: Narrow search results to specific subsets of data
Track Data Quality: Monitor record counts and error rates per dataset
Manage Data Lifecycle: Handle imports, exports, and deletions at the dataset level

Before You Begin

To use the Datasets API, you need:

Create an API Key

In the Pretectum app, go to Configuration → API Keys and create a key. It is shown once, at creation, so copy it then. See API Keys.

Get Business Area and Schema IDs

Retrieve the business area ID using List Business Areas and the schema ID using List Schemas. Dataset queries require both IDs.

Retrieving Datasets

Datasets are accessed through their parent schema. You need both the business area ID and schema ID to list datasets.

Step 1: Get Business Area ID

async function getBusinessAreaId(apiKey, businessAreaName) {
  const response = await fetch('https://api.pretectum.io/v1/my/businessareas', {
    headers: { 'Authorization': apiKey }
  });

  const areas = await response.json();
  const area = areas.find(a => a.name === businessAreaName);
  return area?.businessAreaId;
}

const businessAreaId = await getBusinessAreaId(apiKey, 'Customer');

def get_business_area_id(api_key, business_area_name):
    response = requests.get(
        'https://api.pretectum.io/v1/my/businessareas',
        headers={'Authorization': api_key}
    )
    areas = response.json()

    for area in areas:
        if area['name'] == business_area_name:
            return area['businessAreaId']
    return None

business_area_id = get_business_area_id(api_key, 'Customer')

Step 2: Get Schema ID

async function getSchemaId(apiKey, businessAreaId, schemaName) {
  const response = await fetch(
    `https://api.pretectum.io/v1/businessareas/${businessAreaId}/schemas`,
    {
      headers: { 'Authorization': apiKey }
    }
  );

  const data = await response.json();
  const schema = data.items.find(s => s.name === schemaName);
  return schema?.schemaId;
}

const schemaId = await getSchemaId(apiKey, businessAreaId, 'Individual Customer');

def get_schema_id(api_key, business_area_id, schema_name):
    response = requests.get(
        f'https://api.pretectum.io/v1/businessareas/{business_area_id}/schemas',
        headers={'Authorization': api_key}
    )
    data = response.json()

    for schema in data['items']:
        if schema['name'] == schema_name:
            return schema['schemaId']
    return None

schema_id = get_schema_id(api_key, business_area_id, 'Individual Customer')

Step 3: List Datasets

curl -X GET "https://api.pretectum.io/v1/businessareas/{businessAreaId}/schemas/{schemaId}/datasets" \
  -H "Authorization: pre_your_api_key"

async function getDatasets(apiKey, businessAreaId, schemaId, pageKey = null) {
  const url = new URL(
    `https://api.pretectum.io/v1/businessareas/${businessAreaId}/schemas/${schemaId}/datasets`
  );
  if (pageKey) {
    url.searchParams.set('pageKey', pageKey);
  }

  const response = await fetch(url, {
    headers: {
      'Authorization': apiKey,
      'Accept': 'application/json'
    }
  });

  return response.json();
}

const datasets = await getDatasets(apiKey, businessAreaId, schemaId);
console.log('Available datasets:');
datasets.items.forEach(ds => {
  console.log(`- ${ds.dataSetName}: ${ds.recordCount} records`);
});

def get_datasets(api_key, business_area_id, schema_id, page_key=None):
    params = {}
    if page_key:
        params['pageKey'] = page_key

    response = requests.get(
        f'https://api.pretectum.io/v1/businessareas/{business_area_id}/schemas/{schema_id}/datasets',
        params=params,
        headers={
            'Authorization': api_key,
            'Accept': 'application/json'
        }
    )
    response.raise_for_status()
    return response.json()

datasets = get_datasets(api_key, business_area_id, schema_id)
print('Available datasets:')
for ds in datasets['items']:
    print(f"- {ds['dataSetName']}: {ds['recordCount']} records")

Understanding the Response

{
  "items": [
    {
      "dataSetId": "20240925152201042a1b2c3d4e5f6789012345678901234",
      "dataSetName": "US Customers",
      "dataSetDescription": "Customer records for United States region",
      "businessAreaId": "20240115103000123a1b2c3d4e5f6789012345678901234",
      "businessAreaName": "Customer",
      "schemaId": "20240115103000456d1e2f3a4b5c6789012345678901234",
      "schemaName": "Individual Customer",
      "recordCount": 15420,
      "erroredRecordsCount": 12,
      "runningJobsCount": 0,
      "version": 5,
      "createdByEmail": "admin@example.com",
      "createdByName": "John Admin",
      "updatedByEmail": "admin@example.com",
      "updatedByName": "John Admin",
      "createdDate": "2024-09-25T15:22:01.042Z",
      "updatedDate": "2024-12-15T10:30:00.000Z",
      "deleted": false
    }
  ],
  "nextPageKey": "eyJMYXN0RXZhbHVhdGVkS2V5Ijp7..."
}

Field	Description
`dataSetId`	Unique identifier for the dataset
`dataSetName`	Display name used for filtering in search operations
`dataSetDescription`	Explanation of the dataset’s purpose
`businessAreaId` / `businessAreaName`	Parent business area
`schemaId` / `schemaName`	Parent schema defining the data structure
`recordCount`	Total number of data objects in the dataset
`erroredRecordsCount`	Number of records with validation errors
`runningJobsCount`	Number of background jobs currently processing
`version`	Version number for tracking changes
`deleted`	Whether the dataset has been soft-deleted
`nextPageKey`	Pagination token for fetching the next page

Using Datasets for Search

Once you have the list of datasets, use the dataSetName field to filter your data object searches.

Searching Within a Dataset

# Search within a specific dataset
curl -X GET "https://api.pretectum.io/v1/dataobjects/search?query=John&dataSet=US%20Customers" \
  -H "Authorization: pre_your_api_key"

# Combine with business area and schema filters
curl -X GET "https://api.pretectum.io/v1/dataobjects/search?query=John&businessArea=Customer&schema=Individual%20Customer&dataSet=US%20Customers" \
  -H "Authorization: pre_your_api_key"

async function searchInDataset(apiKey, query, dataSetName, options = {}) {
  const params = new URLSearchParams({
    query,
    dataSet: dataSetName,
    ...options
  });

  const response = await fetch(
    `https://api.pretectum.io/v1/dataobjects/search?${params}`,
    {
      headers: { 'Authorization': apiKey }
    }
  );

  return response.json();
}

// Search for "John" in the US Customers dataset
const results = await searchInDataset(apiKey, 'John', 'US Customers');
console.log(`Found ${results.total} results in US Customers`);

def search_in_dataset(api_key, query, dataset_name, **options):
    params = {
        'query': query,
        'dataSet': dataset_name,
        **options
    }

    response = requests.get(
        'https://api.pretectum.io/v1/dataobjects/search',
        params=params,
        headers={'Authorization': api_key}
    )
    return response.json()

# Search for "John" in the US Customers dataset
results = search_in_dataset(api_key, 'John', 'US Customers')
print(f"Found {results['total']} results in US Customers")

Complete Client Implementation

Here is a complete implementation that handles the full hierarchy from business areas to datasets:

const API_BASE = 'https://api.pretectum.io';

class PretectumClient {
  constructor(apiKey) {
    this.apiKey = apiKey;
    this._businessAreas = null;
    this._schemas = {};
    this._datasets = {};
  }

  async getBusinessAreas() {
    if (this._businessAreas) return this._businessAreas;

    const response = await fetch(`${API_BASE}/v1/my/businessareas`, {
      headers: { 'Authorization': this.apiKey }
    });
    this._businessAreas = await response.json();
    return this._businessAreas;
  }

  async getBusinessAreaId(name) {
    const areas = await this.getBusinessAreas();
    const area = areas.find(a => a.name === name);
    return area?.businessAreaId;
  }

  async getSchemas(businessAreaId) {
    if (this._schemas[businessAreaId]) return this._schemas[businessAreaId];

    const allSchemas = [];
    let pageKey = null;

    do {
      const url = new URL(`${API_BASE}/v1/businessareas/${businessAreaId}/schemas`);
      if (pageKey) url.searchParams.set('pageKey', pageKey);

      const response = await fetch(url, {
        headers: { 'Authorization': this.apiKey }
      });
      const data = await response.json();
      allSchemas.push(...data.items);
      pageKey = data.nextPageKey;
    } while (pageKey);

    this._schemas[businessAreaId] = allSchemas;
    return allSchemas;
  }

  async getSchemaId(businessAreaId, schemaName) {
    const schemas = await this.getSchemas(businessAreaId);
    const schema = schemas.find(s => s.name === schemaName);
    return schema?.schemaId;
  }

  async getDatasets(businessAreaId, schemaId) {
    const cacheKey = `${businessAreaId}_${schemaId}`;
    if (this._datasets[cacheKey]) return this._datasets[cacheKey];

    const allDatasets = [];
    let pageKey = null;

    do {
      const url = new URL(
        `${API_BASE}/v1/businessareas/${businessAreaId}/schemas/${schemaId}/datasets`
      );
      if (pageKey) url.searchParams.set('pageKey', pageKey);

      const response = await fetch(url, {
        headers: { 'Authorization': this.apiKey }
      });
      const data = await response.json();
      allDatasets.push(...data.items);
      pageKey = data.nextPageKey;
    } while (pageKey);

    this._datasets[cacheKey] = allDatasets;
    return allDatasets;
  }

  async getDatasetNames(businessAreaName, schemaName) {
    const businessAreaId = await this.getBusinessAreaId(businessAreaName);
    if (!businessAreaId) throw new Error(`Business area not found: ${businessAreaName}`);

    const schemaId = await this.getSchemaId(businessAreaId, schemaName);
    if (!schemaId) throw new Error(`Schema not found: ${schemaName}`);

    const datasets = await this.getDatasets(businessAreaId, schemaId);
    return datasets
      .filter(ds => !ds.deleted)
      .map(ds => ds.dataSetName);
  }

  async search(query, options = {}) {
    const params = new URLSearchParams({ query, ...options });
    const response = await fetch(`${API_BASE}/dataobjects/search?${params}`, {
      headers: { 'Authorization': this.apiKey }
    });

    return response.json();
  }

  async getDatasetStats(businessAreaName, schemaName) {
    const businessAreaId = await this.getBusinessAreaId(businessAreaName);
    const schemaId = await this.getSchemaId(businessAreaId, schemaName);
    const datasets = await this.getDatasets(businessAreaId, schemaId);

    return {
      totalDatasets: datasets.filter(ds => !ds.deleted).length,
      totalRecords: datasets.reduce((sum, ds) => sum + (ds.recordCount || 0), 0),
      totalErrors: datasets.reduce((sum, ds) => sum + (ds.erroredRecordsCount || 0), 0),
      datasets: datasets.map(ds => ({
        name: ds.dataSetName,
        records: ds.recordCount,
        errors: ds.erroredRecordsCount,
        lastUpdated: ds.updatedDate
      }))
    };
  }
}

// Usage
const client = new PretectumClient(process.env.PRETECTUM_API_KEY);

// Get dataset names for a dropdown
const datasetNames = await client.getDatasetNames('Customer', 'Individual Customer');
console.log('Available datasets:', datasetNames);

// Get statistics
const stats = await client.getDatasetStats('Customer', 'Individual Customer');
console.log(`Total records: ${stats.totalRecords}`);
console.log(`Total errors: ${stats.totalErrors}`);

// Search within a dataset
const results = await client.search('John', {
  businessArea: 'Customer',
  schema: 'Individual Customer',
  dataSet: 'US Customers'
});
console.log(`Found ${results.total} results`);

import os
import requests
from typing import Optional, Dict, List, Any

class PretectumClient:
    API_BASE = 'https://api.pretectum.io'

    def __init__(self, api_key: str):
        self.api_key = api_key
        self._business_areas: Optional[List[Dict]] = None
        self._schemas: Dict[str, List[Dict]] = {}
        self._datasets: Dict[str, List[Dict]] = {}

    def get_business_areas(self) -> List[Dict]:
        if self._business_areas:
            return self._business_areas

        response = requests.get(
            f'{self.API_BASE}/v1/my/businessareas',
            headers={'Authorization': self.api_key}
        )
        response.raise_for_status()
        self._business_areas = response.json()
        return self._business_areas

    def get_business_area_id(self, name: str) -> Optional[str]:
        areas = self.get_business_areas()
        for area in areas:
            if area['name'] == name:
                return area['businessAreaId']
        return None

    def get_schemas(self, business_area_id: str) -> List[Dict]:
        if business_area_id in self._schemas:
            return self._schemas[business_area_id]

        all_schemas = []
        page_key = None

        while True:
            params = {}
            if page_key:
                params['pageKey'] = page_key

            response = requests.get(
                f'{self.API_BASE}/v1/businessareas/{business_area_id}/schemas',
                params=params,
                headers={'Authorization': self.api_key}
            )
            response.raise_for_status()
            data = response.json()
            all_schemas.extend(data['items'])
            page_key = data.get('nextPageKey')
            if not page_key:
                break

        self._schemas[business_area_id] = all_schemas
        return all_schemas

    def get_schema_id(self, business_area_id: str, schema_name: str) -> Optional[str]:
        schemas = self.get_schemas(business_area_id)
        for schema in schemas:
            if schema['name'] == schema_name:
                return schema['schemaId']
        return None

    def get_datasets(self, business_area_id: str, schema_id: str) -> List[Dict]:
        cache_key = f'{business_area_id}_{schema_id}'
        if cache_key in self._datasets:
            return self._datasets[cache_key]

        all_datasets = []
        page_key = None

        while True:
            params = {}
            if page_key:
                params['pageKey'] = page_key

            response = requests.get(
                f'{self.API_BASE}/v1/businessareas/{business_area_id}/schemas/{schema_id}/datasets',
                params=params,
                headers={'Authorization': self.api_key}
            )
            response.raise_for_status()
            data = response.json()
            all_datasets.extend(data['items'])
            page_key = data.get('nextPageKey')
            if not page_key:
                break

        self._datasets[cache_key] = all_datasets
        return all_datasets

    def get_dataset_names(self, business_area_name: str, schema_name: str) -> List[str]:
        business_area_id = self.get_business_area_id(business_area_name)
        if not business_area_id:
            raise ValueError(f"Business area not found: {business_area_name}")

        schema_id = self.get_schema_id(business_area_id, schema_name)
        if not schema_id:
            raise ValueError(f"Schema not found: {schema_name}")

        datasets = self.get_datasets(business_area_id, schema_id)
        return [ds['dataSetName'] for ds in datasets if not ds.get('deleted', False)]

    def search(self, query: str, **options) -> Dict[str, Any]:
        params = {'query': query, **options}
        response = requests.get(
            f'{self.API_BASE}/dataobjects/search',
            params=params,
            headers={'Authorization': self.api_key}
        )
        response.raise_for_status()
        return response.json()

    def get_dataset_stats(self, business_area_name: str, schema_name: str) -> Dict:
        business_area_id = self.get_business_area_id(business_area_name)
        schema_id = self.get_schema_id(business_area_id, schema_name)
        datasets = self.get_datasets(business_area_id, schema_id)

        active_datasets = [ds for ds in datasets if not ds.get('deleted', False)]

        return {
            'total_datasets': len(active_datasets),
            'total_records': sum(ds.get('recordCount', 0) for ds in active_datasets),
            'total_errors': sum(ds.get('erroredRecordsCount', 0) for ds in active_datasets),
            'datasets': [
                {
                    'name': ds['dataSetName'],
                    'records': ds.get('recordCount', 0),
                    'errors': ds.get('erroredRecordsCount', 0),
                    'last_updated': ds.get('updatedDate')
                }
                for ds in active_datasets
            ]
        }


# Usage
client = PretectumClient(os.environ['PRETECTUM_API_KEY'])

# Get dataset names for a dropdown
dataset_names = client.get_dataset_names('Customer', 'Individual Customer')
print('Available datasets:', dataset_names)

# Get statistics
stats = client.get_dataset_stats('Customer', 'Individual Customer')
print(f"Total records: {stats['total_records']}")
print(f"Total errors: {stats['total_errors']}")

# Search within a dataset
results = client.search('John',
    businessArea='Customer',
    schema='Individual Customer',
    dataSet='US Customers'
)
print(f"Found {results['total']} results")

Building a Dataset Selector

Create a cascading filter interface for business area → schema → dataset:

import { useState, useEffect } from 'react';

function DatasetSelector({ client, onSelect }) {
  const [businessAreas, setBusinessAreas] = useState([]);
  const [schemas, setSchemas] = useState([]);
  const [datasets, setDatasets] = useState([]);
  const [selectedArea, setSelectedArea] = useState('');
  const [selectedSchema, setSelectedSchema] = useState('');
  const [selectedDataset, setSelectedDataset] = useState('');

  // Load business areas on mount
  useEffect(() => {
    client.getBusinessAreas().then(areas => {
      setBusinessAreas(areas.filter(a => a.active));
    });
  }, [client]);

  // Load schemas when business area changes
  useEffect(() => {
    if (!selectedArea) {
      setSchemas([]);
      setSelectedSchema('');
      return;
    }

    const areaId = businessAreas.find(a => a.name === selectedArea)?.businessAreaId;
    if (areaId) {
      client.getSchemas(areaId).then(schemas => {
        setSchemas(schemas.filter(s => s.active));
        setSelectedSchema('');
      });
    }
  }, [selectedArea, businessAreas, client]);

  // Load datasets when schema changes
  useEffect(() => {
    if (!selectedArea || !selectedSchema) {
      setDatasets([]);
      setSelectedDataset('');
      return;
    }

    const areaId = businessAreas.find(a => a.name === selectedArea)?.businessAreaId;
    const schemaId = schemas.find(s => s.name === selectedSchema)?.schemaId;

    if (areaId && schemaId) {
      client.getDatasets(areaId, schemaId).then(datasets => {
        setDatasets(datasets.filter(ds => !ds.deleted));
        setSelectedDataset('');
      });
    }
  }, [selectedSchema, selectedArea, schemas, businessAreas, client]);

  // Notify parent of selection change
  useEffect(() => {
    onSelect({
      businessArea: selectedArea || null,
      schema: selectedSchema || null,
      dataSet: selectedDataset || null
    });
  }, [selectedArea, selectedSchema, selectedDataset, onSelect]);

  return (
    <div className="dataset-selector">
      <select value={selectedArea} onChange={e => setSelectedArea(e.target.value)}>
        <option value="">All Business Areas</option>
        {businessAreas.map(area => (
          <option key={area.businessAreaId} value={area.name}>{area.name}</option>
        ))}
      </select>

      <select
        value={selectedSchema}
        onChange={e => setSelectedSchema(e.target.value)}
        disabled={!selectedArea}
      >
        <option value="">All Schemas</option>
        {schemas.map(schema => (
          <option key={schema.schemaId} value={schema.name}>{schema.name}</option>
        ))}
      </select>

      <select
        value={selectedDataset}
        onChange={e => setSelectedDataset(e.target.value)}
        disabled={!selectedSchema}
      >
        <option value="">All Datasets</option>
        {datasets.map(ds => (
          <option key={ds.dataSetId} value={ds.dataSetName}>
            {ds.dataSetName} ({ds.recordCount} records)
          </option>
        ))}
      </select>
    </div>
  );
}

Best Practices

Cache Dataset Metadata

Dataset metadata changes less frequently than actual data. Cache the list and refresh periodically:

const CACHE_TTL = 5 * 60 * 1000; // 5 minutes
let cachedAt = 0;

async function getCachedDatasets(businessAreaId, schemaId) {
  const cacheKey = `${businessAreaId}_${schemaId}`;
  if (Date.now() - cachedAt > CACHE_TTL) {
    client._datasets[cacheKey] = null;
    cachedAt = Date.now();
  }
  return client.getDatasets(businessAreaId, schemaId);
}

Filter Active Datasets

Always filter out deleted datasets in user-facing interfaces:

active_datasets = [
    ds for ds in datasets['items']
    if not ds.get('deleted', False)
]

Monitor Data Quality

Regularly check error counts to identify data quality issues:

const datasetsWithErrors = datasets.filter(ds => ds.erroredRecordsCount > 0);
if (datasetsWithErrors.length > 0) {
  console.warn('Datasets with errors:', datasetsWithErrors.map(ds => ds.dataSetName));
}

Use Names for Search Filters

When filtering searches, use the dataSetName, not the dataSetId:

# Correct - use name
?dataSet=US%20Customers

# Incorrect - don't use ID
?dataSet=20240925152201042a1b2c3d4e5f6789012345678901234

Next Steps

List Datasets API

View the complete List Datasets API reference

Search Data Objects

Learn how to search within datasets

Schemas Guide

Learn about schemas

Business Areas Guide

Learn about business areas

Getting Started

Business Areas

Schemas

Datasets

Data Objects

AI Tools

Working with Datasets

Overview

Why Datasets Matter

Before You Begin

Retrieving Datasets

Step 1: Get Business Area ID

Step 2: Get Schema ID

Step 3: List Datasets

Understanding the Response

Using Datasets for Search

Searching Within a Dataset

Complete Client Implementation

Building a Dataset Selector

Best Practices

Next Steps

List Datasets API

Search Data Objects

Schemas Guide

Business Areas Guide

​Overview

​Why Datasets Matter

​Before You Begin

​Retrieving Datasets

​Step 1: Get Business Area ID

​Step 2: Get Schema ID

​Step 3: List Datasets

​Understanding the Response

​Using Datasets for Search

​Searching Within a Dataset

​Complete Client Implementation

​Building a Dataset Selector

​Best Practices

​Next Steps

List Datasets API

Search Data Objects

Schemas Guide

Business Areas Guide

Overview

Why Datasets Matter

Before You Begin

Retrieving Datasets

Step 1: Get Business Area ID

Step 2: Get Schema ID

Step 3: List Datasets

Understanding the Response

Using Datasets for Search

Searching Within a Dataset

Complete Client Implementation

Building a Dataset Selector

Best Practices

Next Steps