Working with Large Data

When working with API resources, you may encounter scenarios where responses contain large amounts of data. The Oystehr API has a 6 MB response size limit. Responses exceeding this limit will result in a 413 Payload Too Large error.

When a response exceeds this limit, you will receive an error payload similar to this:

{
  "code": "PAYLOAD_TOO_LARGE",
  "message": "The response size (7.82 MB) exceeds the maximum allowed size of 6 MB. Please refine your query with pagination or use the _elements parameter for FHIR resources.",
  "documentationUrl": "https://docs.oystehr.com/oystehr/core-documentation/working-with-large-data"
}

This document provides strategies for efficiently working with large datasets, with specific emphasis on FHIR resources.

Strategies for Working with Large Data

1. Use Pagination

The most common approach to handling large datasets is pagination. For FHIR resources, search operations return a Bundle that supports pagination through the _count and _offset parameters:

import { Patient } from 'fhir/r4b';
import Oystehr from '@oystehr/sdk';
 
const oystehr = new Oystehr({
  accessToken: "<your_access_token>",
});
 
// Fetch first 50 patients
const firstPage = await oystehr.fhir.search<Patient>({
  resourceType: 'Patient',
  params: [
    { name: '_count', value: '50' },
    { name: '_offset', value: '0' },
  ],
});
 
// Fetch next 50 patients
const secondPage = await oystehr.fhir.search<Patient>({
  resourceType: 'Patient',
  params: [
    { name: '_count', value: '50' },
    { name: '_offset', value: '50' },
  ],
});

You can also follow the pagination links in the Bundle response:

{
  "resourceType": "Bundle",
  "type": "searchset",
  "link": [
    {
      "relation": "self",
      "url": "https://fhir-api.zapehr.com/r4/Patient?_count=50&_offset=0"
    },
    {
      "relation": "next",
      "url": "https://fhir-api.zapehr.com/r4/Patient?_count=50&_offset=50"
    }
  ],
  "entry": [
    // resources...
  ]
}

2. Use the _elements Parameter (FHIR)

For FHIR resources, the _elements parameter allows you to specify which elements should be included in the response. This is particularly useful when you only need specific fields from large resources.

import { Patient } from 'fhir/r4b';
import Oystehr from '@oystehr/sdk';
 
const oystehr = new Oystehr({
  accessToken: "<your_access_token>",
});
 
// Only retrieve id, name, and birthDate fields
const patientBundle = await oystehr.fhir.search<Patient>({
  resourceType: 'Patient',
  params: [
    { name: '_elements', value: 'id,name,birthDate' },
  ],
});

When working with resources that contain large arrays, using _elements can dramatically reduce response size.

3. Limit Included Resources

The _include parameter lets you retrieve related resources in one request, but it can significantly increase response size. Use it selectively and combine with _elements when needed:

import { Encounter } from 'fhir/r4b';
import Oystehr from '@oystehr/sdk';
 
const oystehr = new Oystehr({
  accessToken: "<your_access_token>",
});
 
// Only include essential fields
const encounters = await oystehr.fhir.search<Encounter>({
  resourceType: 'Encounter',
  params: [
    { name: '_include', value: 'Encounter:patient' },
    { name: '_elements', value: 'id,status,period' },
  ],
});