> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pretectum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Working with Datasets

> Learn how to retrieve and use datasets to organize and filter your master data

This guide explains how to work with datasets in Pretectum using the API. You will learn how to retrieve datasets, understand their properties, and use them effectively for organizing and searching your data.

## Overview

Datasets are collections of data objects (records) that share the same schema structure. They represent the actual data stored in your Pretectum master data repository and provide a way to organize records into logical groups.

The hierarchy in Pretectum is:

```
Business Area (e.g., Customer)
  └── Schema (e.g., Individual Customer)
       └── Dataset (e.g., US Customers, European Customers)
            └── Data Objects (actual customer records)
```

### Why Datasets Matter

Datasets help you:

1. **Organize Data**: Group records by region, source, time period, or any logical category
2. **Filter Searches**: Narrow search results to specific subsets of data
3. **Track Data Quality**: Monitor record counts and error rates per dataset
4. **Manage Data Lifecycle**: Handle imports, exports, and deletions at the dataset level

## Before You Begin

To use the Datasets API, you need:

<Steps>
  <Step title="Create an API Key">
    In the Pretectum app, go to **Configuration → API Keys** and create a key. It is shown once, at
    creation, so copy it then. See [API Keys](/api-reference/authentication/api-keys).
  </Step>

  <Step title="Get Business Area and Schema IDs">
    Retrieve the business area ID using [List Business Areas](/api-reference/business-areas/list) and the schema ID using [List Schemas](/api-reference/schemas/list). Dataset queries require both IDs.
  </Step>
</Steps>

## Retrieving Datasets

Datasets are accessed through their parent schema. You need both the business area ID and schema ID to list datasets.

### Step 1: Get Business Area ID

<CodeGroup>
  ```javascript JavaScript theme={null}
  async function getBusinessAreaId(apiKey, businessAreaName) {
    const response = await fetch('https://api.pretectum.io/v1/my/businessareas', {
      headers: { 'Authorization': apiKey }
    });

    const areas = await response.json();
    const area = areas.find(a => a.name === businessAreaName);
    return area?.businessAreaId;
  }

  const businessAreaId = await getBusinessAreaId(apiKey, 'Customer');
  ```

  ```python Python theme={null}
  def get_business_area_id(api_key, business_area_name):
      response = requests.get(
          'https://api.pretectum.io/v1/my/businessareas',
          headers={'Authorization': api_key}
      )
      areas = response.json()

      for area in areas:
          if area['name'] == business_area_name:
              return area['businessAreaId']
      return None

  business_area_id = get_business_area_id(api_key, 'Customer')
  ```
</CodeGroup>

### Step 2: Get Schema ID

<CodeGroup>
  ```javascript JavaScript theme={null}
  async function getSchemaId(apiKey, businessAreaId, schemaName) {
    const response = await fetch(
      `https://api.pretectum.io/v1/businessareas/${businessAreaId}/schemas`,
      {
        headers: { 'Authorization': apiKey }
      }
    );

    const data = await response.json();
    const schema = data.items.find(s => s.name === schemaName);
    return schema?.schemaId;
  }

  const schemaId = await getSchemaId(apiKey, businessAreaId, 'Individual Customer');
  ```

  ```python Python theme={null}
  def get_schema_id(api_key, business_area_id, schema_name):
      response = requests.get(
          f'https://api.pretectum.io/v1/businessareas/{business_area_id}/schemas',
          headers={'Authorization': api_key}
      )
      data = response.json()

      for schema in data['items']:
          if schema['name'] == schema_name:
              return schema['schemaId']
      return None

  schema_id = get_schema_id(api_key, business_area_id, 'Individual Customer')
  ```
</CodeGroup>

### Step 3: List Datasets

<CodeGroup>
  ```bash cURL theme={null}
  curl -X GET "https://api.pretectum.io/v1/businessareas/{businessAreaId}/schemas/{schemaId}/datasets" \
    -H "Authorization: pre_your_api_key"
  ```

  ```javascript JavaScript theme={null}
  async function getDatasets(apiKey, businessAreaId, schemaId, pageKey = null) {
    const url = new URL(
      `https://api.pretectum.io/v1/businessareas/${businessAreaId}/schemas/${schemaId}/datasets`
    );
    if (pageKey) {
      url.searchParams.set('pageKey', pageKey);
    }

    const response = await fetch(url, {
      headers: {
        'Authorization': apiKey,
        'Accept': 'application/json'
      }
    });

    return response.json();
  }

  const datasets = await getDatasets(apiKey, businessAreaId, schemaId);
  console.log('Available datasets:');
  datasets.items.forEach(ds => {
    console.log(`- ${ds.dataSetName}: ${ds.recordCount} records`);
  });
  ```

  ```python Python theme={null}
  def get_datasets(api_key, business_area_id, schema_id, page_key=None):
      params = {}
      if page_key:
          params['pageKey'] = page_key

      response = requests.get(
          f'https://api.pretectum.io/v1/businessareas/{business_area_id}/schemas/{schema_id}/datasets',
          params=params,
          headers={
              'Authorization': api_key,
              'Accept': 'application/json'
          }
      )
      response.raise_for_status()
      return response.json()

  datasets = get_datasets(api_key, business_area_id, schema_id)
  print('Available datasets:')
  for ds in datasets['items']:
      print(f"- {ds['dataSetName']}: {ds['recordCount']} records")
  ```
</CodeGroup>

### Understanding the Response

```json theme={null}
{
  "items": [
    {
      "dataSetId": "20240925152201042a1b2c3d4e5f6789012345678901234",
      "dataSetName": "US Customers",
      "dataSetDescription": "Customer records for United States region",
      "businessAreaId": "20240115103000123a1b2c3d4e5f6789012345678901234",
      "businessAreaName": "Customer",
      "schemaId": "20240115103000456d1e2f3a4b5c6789012345678901234",
      "schemaName": "Individual Customer",
      "recordCount": 15420,
      "erroredRecordsCount": 12,
      "runningJobsCount": 0,
      "version": 5,
      "createdByEmail": "admin@example.com",
      "createdByName": "John Admin",
      "updatedByEmail": "admin@example.com",
      "updatedByName": "John Admin",
      "createdDate": "2024-09-25T15:22:01.042Z",
      "updatedDate": "2024-12-15T10:30:00.000Z",
      "deleted": false
    }
  ],
  "nextPageKey": "eyJMYXN0RXZhbHVhdGVkS2V5Ijp7..."
}
```

| Field | Description |
| - | - |
| `dataSetId` | Unique identifier for the dataset |
| `dataSetName` | Display name used for filtering in search operations |
| `dataSetDescription` | Explanation of the dataset's purpose |
| `businessAreaId` / `businessAreaName` | Parent business area |
| `schemaId` / `schemaName` | Parent schema defining the data structure |
| `recordCount` | Total number of data objects in the dataset |
| `erroredRecordsCount` | Number of records with validation errors |
| `runningJobsCount` | Number of background jobs currently processing |
| `version` | Version number for tracking changes |
| `deleted` | Whether the dataset has been soft-deleted |
| `nextPageKey` | Pagination token for fetching the next page |

## Using Datasets for Search

Once you have the list of datasets, use the `dataSetName` field to filter your data object searches.

### Searching Within a Dataset

<CodeGroup>
  ```bash cURL theme={null}
  # Search within a specific dataset
  curl -X GET "https://api.pretectum.io/v1/dataobjects/search?query=John&dataSet=US%20Customers" \
    -H "Authorization: pre_your_api_key"

  # Combine with business area and schema filters
  curl -X GET "https://api.pretectum.io/v1/dataobjects/search?query=John&businessArea=Customer&schema=Individual%20Customer&dataSet=US%20Customers" \
    -H "Authorization: pre_your_api_key"
  ```

  ```javascript JavaScript theme={null}
  async function searchInDataset(apiKey, query, dataSetName, options = {}) {
    const params = new URLSearchParams({
      query,
      dataSet: dataSetName,
      ...options
    });

    const response = await fetch(
      `https://api.pretectum.io/v1/dataobjects/search?${params}`,
      {
        headers: { 'Authorization': apiKey }
      }
    );

    return response.json();
  }

  // Search for "John" in the US Customers dataset
  const results = await searchInDataset(apiKey, 'John', 'US Customers');
  console.log(`Found ${results.total} results in US Customers`);
  ```

  ```python Python theme={null}
  def search_in_dataset(api_key, query, dataset_name, **options):
      params = {
          'query': query,
          'dataSet': dataset_name,
          **options
      }

      response = requests.get(
          'https://api.pretectum.io/v1/dataobjects/search',
          params=params,
          headers={'Authorization': api_key}
      )
      return response.json()

  # Search for "John" in the US Customers dataset
  results = search_in_dataset(api_key, 'John', 'US Customers')
  print(f"Found {results['total']} results in US Customers")
  ```
</CodeGroup>

## Complete Client Implementation

Here is a complete implementation that handles the full hierarchy from business areas to datasets:

<CodeGroup>
  ```javascript JavaScript theme={null}
  const API_BASE = 'https://api.pretectum.io';

  class PretectumClient {
    constructor(apiKey) {
      this.apiKey = apiKey;
      this._businessAreas = null;
      this._schemas = {};
      this._datasets = {};
    }

    async getBusinessAreas() {
      if (this._businessAreas) return this._businessAreas;

      const response = await fetch(`${API_BASE}/v1/my/businessareas`, {
        headers: { 'Authorization': this.apiKey }
      });
      this._businessAreas = await response.json();
      return this._businessAreas;
    }

    async getBusinessAreaId(name) {
      const areas = await this.getBusinessAreas();
      const area = areas.find(a => a.name === name);
      return area?.businessAreaId;
    }

    async getSchemas(businessAreaId) {
      if (this._schemas[businessAreaId]) return this._schemas[businessAreaId];

      const allSchemas = [];
      let pageKey = null;

      do {
        const url = new URL(`${API_BASE}/v1/businessareas/${businessAreaId}/schemas`);
        if (pageKey) url.searchParams.set('pageKey', pageKey);

        const response = await fetch(url, {
          headers: { 'Authorization': this.apiKey }
        });
        const data = await response.json();
        allSchemas.push(...data.items);
        pageKey = data.nextPageKey;
      } while (pageKey);

      this._schemas[businessAreaId] = allSchemas;
      return allSchemas;
    }

    async getSchemaId(businessAreaId, schemaName) {
      const schemas = await this.getSchemas(businessAreaId);
      const schema = schemas.find(s => s.name === schemaName);
      return schema?.schemaId;
    }

    async getDatasets(businessAreaId, schemaId) {
      const cacheKey = `${businessAreaId}_${schemaId}`;
      if (this._datasets[cacheKey]) return this._datasets[cacheKey];

      const allDatasets = [];
      let pageKey = null;

      do {
        const url = new URL(
          `${API_BASE}/v1/businessareas/${businessAreaId}/schemas/${schemaId}/datasets`
        );
        if (pageKey) url.searchParams.set('pageKey', pageKey);

        const response = await fetch(url, {
          headers: { 'Authorization': this.apiKey }
        });
        const data = await response.json();
        allDatasets.push(...data.items);
        pageKey = data.nextPageKey;
      } while (pageKey);

      this._datasets[cacheKey] = allDatasets;
      return allDatasets;
    }

    async getDatasetNames(businessAreaName, schemaName) {
      const businessAreaId = await this.getBusinessAreaId(businessAreaName);
      if (!businessAreaId) throw new Error(`Business area not found: ${businessAreaName}`);

      const schemaId = await this.getSchemaId(businessAreaId, schemaName);
      if (!schemaId) throw new Error(`Schema not found: ${schemaName}`);

      const datasets = await this.getDatasets(businessAreaId, schemaId);
      return datasets
        .filter(ds => !ds.deleted)
        .map(ds => ds.dataSetName);
    }

    async search(query, options = {}) {
      const params = new URLSearchParams({ query, ...options });
      const response = await fetch(`${API_BASE}/dataobjects/search?${params}`, {
        headers: { 'Authorization': this.apiKey }
      });

      return response.json();
    }

    async getDatasetStats(businessAreaName, schemaName) {
      const businessAreaId = await this.getBusinessAreaId(businessAreaName);
      const schemaId = await this.getSchemaId(businessAreaId, schemaName);
      const datasets = await this.getDatasets(businessAreaId, schemaId);

      return {
        totalDatasets: datasets.filter(ds => !ds.deleted).length,
        totalRecords: datasets.reduce((sum, ds) => sum + (ds.recordCount || 0), 0),
        totalErrors: datasets.reduce((sum, ds) => sum + (ds.erroredRecordsCount || 0), 0),
        datasets: datasets.map(ds => ({
          name: ds.dataSetName,
          records: ds.recordCount,
          errors: ds.erroredRecordsCount,
          lastUpdated: ds.updatedDate
        }))
      };
    }
  }

  // Usage
  const client = new PretectumClient(process.env.PRETECTUM_API_KEY);

  // Get dataset names for a dropdown
  const datasetNames = await client.getDatasetNames('Customer', 'Individual Customer');
  console.log('Available datasets:', datasetNames);

  // Get statistics
  const stats = await client.getDatasetStats('Customer', 'Individual Customer');
  console.log(`Total records: ${stats.totalRecords}`);
  console.log(`Total errors: ${stats.totalErrors}`);

  // Search within a dataset
  const results = await client.search('John', {
    businessArea: 'Customer',
    schema: 'Individual Customer',
    dataSet: 'US Customers'
  });
  console.log(`Found ${results.total} results`);
  ```

  ```python Python theme={null}
  import os
  import requests
  from typing import Optional, Dict, List, Any

  class PretectumClient:
      API_BASE = 'https://api.pretectum.io'

      def __init__(self, api_key: str):
          self.api_key = api_key
          self._business_areas: Optional[List[Dict]] = None
          self._schemas: Dict[str, List[Dict]] = {}
          self._datasets: Dict[str, List[Dict]] = {}

      def get_business_areas(self) -> List[Dict]:
          if self._business_areas:
              return self._business_areas

          response = requests.get(
              f'{self.API_BASE}/v1/my/businessareas',
              headers={'Authorization': self.api_key}
          )
          response.raise_for_status()
          self._business_areas = response.json()
          return self._business_areas

      def get_business_area_id(self, name: str) -> Optional[str]:
          areas = self.get_business_areas()
          for area in areas:
              if area['name'] == name:
                  return area['businessAreaId']
          return None

      def get_schemas(self, business_area_id: str) -> List[Dict]:
          if business_area_id in self._schemas:
              return self._schemas[business_area_id]

          all_schemas = []
          page_key = None

          while True:
              params = {}
              if page_key:
                  params['pageKey'] = page_key

              response = requests.get(
                  f'{self.API_BASE}/v1/businessareas/{business_area_id}/schemas',
                  params=params,
                  headers={'Authorization': self.api_key}
              )
              response.raise_for_status()
              data = response.json()
              all_schemas.extend(data['items'])
              page_key = data.get('nextPageKey')
              if not page_key:
                  break

          self._schemas[business_area_id] = all_schemas
          return all_schemas

      def get_schema_id(self, business_area_id: str, schema_name: str) -> Optional[str]:
          schemas = self.get_schemas(business_area_id)
          for schema in schemas:
              if schema['name'] == schema_name:
                  return schema['schemaId']
          return None

      def get_datasets(self, business_area_id: str, schema_id: str) -> List[Dict]:
          cache_key = f'{business_area_id}_{schema_id}'
          if cache_key in self._datasets:
              return self._datasets[cache_key]

          all_datasets = []
          page_key = None

          while True:
              params = {}
              if page_key:
                  params['pageKey'] = page_key

              response = requests.get(
                  f'{self.API_BASE}/v1/businessareas/{business_area_id}/schemas/{schema_id}/datasets',
                  params=params,
                  headers={'Authorization': self.api_key}
              )
              response.raise_for_status()
              data = response.json()
              all_datasets.extend(data['items'])
              page_key = data.get('nextPageKey')
              if not page_key:
                  break

          self._datasets[cache_key] = all_datasets
          return all_datasets

      def get_dataset_names(self, business_area_name: str, schema_name: str) -> List[str]:
          business_area_id = self.get_business_area_id(business_area_name)
          if not business_area_id:
              raise ValueError(f"Business area not found: {business_area_name}")

          schema_id = self.get_schema_id(business_area_id, schema_name)
          if not schema_id:
              raise ValueError(f"Schema not found: {schema_name}")

          datasets = self.get_datasets(business_area_id, schema_id)
          return [ds['dataSetName'] for ds in datasets if not ds.get('deleted', False)]

      def search(self, query: str, **options) -> Dict[str, Any]:
          params = {'query': query, **options}
          response = requests.get(
              f'{self.API_BASE}/dataobjects/search',
              params=params,
              headers={'Authorization': self.api_key}
          )
          response.raise_for_status()
          return response.json()

      def get_dataset_stats(self, business_area_name: str, schema_name: str) -> Dict:
          business_area_id = self.get_business_area_id(business_area_name)
          schema_id = self.get_schema_id(business_area_id, schema_name)
          datasets = self.get_datasets(business_area_id, schema_id)

          active_datasets = [ds for ds in datasets if not ds.get('deleted', False)]

          return {
              'total_datasets': len(active_datasets),
              'total_records': sum(ds.get('recordCount', 0) for ds in active_datasets),
              'total_errors': sum(ds.get('erroredRecordsCount', 0) for ds in active_datasets),
              'datasets': [
                  {
                      'name': ds['dataSetName'],
                      'records': ds.get('recordCount', 0),
                      'errors': ds.get('erroredRecordsCount', 0),
                      'last_updated': ds.get('updatedDate')
                  }
                  for ds in active_datasets
              ]
          }


  # Usage
  client = PretectumClient(os.environ['PRETECTUM_API_KEY'])

  # Get dataset names for a dropdown
  dataset_names = client.get_dataset_names('Customer', 'Individual Customer')
  print('Available datasets:', dataset_names)

  # Get statistics
  stats = client.get_dataset_stats('Customer', 'Individual Customer')
  print(f"Total records: {stats['total_records']}")
  print(f"Total errors: {stats['total_errors']}")

  # Search within a dataset
  results = client.search('John',
      businessArea='Customer',
      schema='Individual Customer',
      dataSet='US Customers'
  )
  print(f"Found {results['total']} results")
  ```
</CodeGroup>

## Building a Dataset Selector

Create a cascading filter interface for business area → schema → dataset:

<CodeGroup>
  ```javascript JavaScript (React Example) theme={null}
  import { useState, useEffect } from 'react';

  function DatasetSelector({ client, onSelect }) {
    const [businessAreas, setBusinessAreas] = useState([]);
    const [schemas, setSchemas] = useState([]);
    const [datasets, setDatasets] = useState([]);
    const [selectedArea, setSelectedArea] = useState('');
    const [selectedSchema, setSelectedSchema] = useState('');
    const [selectedDataset, setSelectedDataset] = useState('');

    // Load business areas on mount
    useEffect(() => {
      client.getBusinessAreas().then(areas => {
        setBusinessAreas(areas.filter(a => a.active));
      });
    }, [client]);

    // Load schemas when business area changes
    useEffect(() => {
      if (!selectedArea) {
        setSchemas([]);
        setSelectedSchema('');
        return;
      }

      const areaId = businessAreas.find(a => a.name === selectedArea)?.businessAreaId;
      if (areaId) {
        client.getSchemas(areaId).then(schemas => {
          setSchemas(schemas.filter(s => s.active));
          setSelectedSchema('');
        });
      }
    }, [selectedArea, businessAreas, client]);

    // Load datasets when schema changes
    useEffect(() => {
      if (!selectedArea || !selectedSchema) {
        setDatasets([]);
        setSelectedDataset('');
        return;
      }

      const areaId = businessAreas.find(a => a.name === selectedArea)?.businessAreaId;
      const schemaId = schemas.find(s => s.name === selectedSchema)?.schemaId;

      if (areaId && schemaId) {
        client.getDatasets(areaId, schemaId).then(datasets => {
          setDatasets(datasets.filter(ds => !ds.deleted));
          setSelectedDataset('');
        });
      }
    }, [selectedSchema, selectedArea, schemas, businessAreas, client]);

    // Notify parent of selection change
    useEffect(() => {
      onSelect({
        businessArea: selectedArea || null,
        schema: selectedSchema || null,
        dataSet: selectedDataset || null
      });
    }, [selectedArea, selectedSchema, selectedDataset, onSelect]);

    return (
      <div className="dataset-selector">
        <select value={selectedArea} onChange={e => setSelectedArea(e.target.value)}>
          <option value="">All Business Areas</option>
          {businessAreas.map(area => (
            <option key={area.businessAreaId} value={area.name}>{area.name}</option>
          ))}
        </select>

        <select
          value={selectedSchema}
          onChange={e => setSelectedSchema(e.target.value)}
          disabled={!selectedArea}
        >
          <option value="">All Schemas</option>
          {schemas.map(schema => (
            <option key={schema.schemaId} value={schema.name}>{schema.name}</option>
          ))}
        </select>

        <select
          value={selectedDataset}
          onChange={e => setSelectedDataset(e.target.value)}
          disabled={!selectedSchema}
        >
          <option value="">All Datasets</option>
          {datasets.map(ds => (
            <option key={ds.dataSetId} value={ds.dataSetName}>
              {ds.dataSetName} ({ds.recordCount} records)
            </option>
          ))}
        </select>
      </div>
    );
  }
  ```
</CodeGroup>

## Best Practices

<AccordionGroup>
  <Accordion title="Cache Dataset Metadata">
    Dataset metadata changes less frequently than actual data. Cache the list and refresh periodically:

    ```javascript theme={null}
    const CACHE_TTL = 5 * 60 * 1000; // 5 minutes
    let cachedAt = 0;

    async function getCachedDatasets(businessAreaId, schemaId) {
      const cacheKey = `${businessAreaId}_${schemaId}`;
      if (Date.now() - cachedAt > CACHE_TTL) {
        client._datasets[cacheKey] = null;
        cachedAt = Date.now();
      }
      return client.getDatasets(businessAreaId, schemaId);
    }
    ```
  </Accordion>

  <Accordion title="Filter Active Datasets">
    Always filter out deleted datasets in user-facing interfaces:

    ```python theme={null}
    active_datasets = [
        ds for ds in datasets['items']
        if not ds.get('deleted', False)
    ]
    ```
  </Accordion>

  <Accordion title="Monitor Data Quality">
    Regularly check error counts to identify data quality issues:

    ```javascript theme={null}
    const datasetsWithErrors = datasets.filter(ds => ds.erroredRecordsCount > 0);
    if (datasetsWithErrors.length > 0) {
      console.warn('Datasets with errors:', datasetsWithErrors.map(ds => ds.dataSetName));
    }
    ```
  </Accordion>

  <Accordion title="Use Names for Search Filters">
    When filtering searches, use the `dataSetName`, not the `dataSetId`:

    ```bash theme={null}
    # Correct - use name
    ?dataSet=US%20Customers

    # Incorrect - don't use ID
    ?dataSet=20240925152201042a1b2c3d4e5f6789012345678901234
    ```
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="List Datasets API" icon="list" href="/api-reference/datasets/list">
    View the complete List Datasets API reference
  </Card>

  <Card title="Search Data Objects" icon="magnifying-glass" href="/guides/dataobjects/search">
    Learn how to search within datasets
  </Card>

  <Card title="Schemas Guide" icon="file-code" href="/guides/schemas/overview">
    Learn about schemas
  </Card>

  <Card title="Business Areas Guide" icon="building" href="/guides/business-areas/overview">
    Learn about business areas
  </Card>
</CardGroup>