Working with Large Data

When working with API resources, you may encounter scenarios where responses contain large amounts of data. The Oystehr API has a 6 MiB response size limit (6,291,456 bytes). Responses exceeding this limit will result in a 413 Content Too Large error.

Note that during processing of a response, the data is serialized, internally, in a way that can inflate it to almost four times the actual size. If the data contains large numbers of characters that need to be escaped when serialized, such as backslashes and double quotes, the internal response size can unexpectedly end up exceeding the limit, which triggers the error response.

When a response exceeds the limit, internally, you will receive an error payload similar to this:

{
  "code": "4130",
  "message": "An internal response size (7,003,938) exceeds the maximum allowed size (6,291,456).  Please refine your query with pagination or use the _elements parameter for FHIR resources.  See: https://docs.zapehr.com/oystehr/core-documentation/working-with-large-data"
}

This document provides strategies for efficiently working with large datasets, with specific emphasis on FHIR resources.

Strategies for Working with Large Data

1. Use Pagination

The most common approach to handling large datasets is pagination. For FHIR resources, search operations return a Bundle that supports pagination through the _count and _offset parameters:

import { Patient } from 'fhir/r4b';
import Oystehr from '@oystehr/sdk';
 
const oystehr = new Oystehr({
  accessToken: "<your_access_token>",
});
 
// Fetch first 50 patients
const firstPage = await oystehr.fhir.search<Patient>({
  resourceType: 'Patient',
  params: [
    { name: '_count', value: '50' },
    { name: '_offset', value: '0' },
  ],
});
 
// Fetch next 50 patients
const secondPage = await oystehr.fhir.search<Patient>({
  resourceType: 'Patient',
  params: [
    { name: '_count', value: '50' },
    { name: '_offset', value: '50' },
  ],
});

You can also follow the pagination links in the Bundle response:

{
  "resourceType": "Bundle",
  "type": "searchset",
  "link": [
    {
      "relation": "self",
      "url": "https://fhir-api.zapehr.com/r4/Patient?_count=50&_offset=0"
    },
    {
      "relation": "next",
      "url": "https://fhir-api.zapehr.com/r4/Patient?_count=50&_offset=50"
    }
  ],
  "entry": [
    // resources...
  ]
}

2. Use the _elements Parameter (FHIR)

For FHIR resources, the _elements parameter allows you to specify which elements should be included in the response. This is particularly useful when you only need specific fields from large resources.

import { Patient } from 'fhir/r4b';
import Oystehr from '@oystehr/sdk';
 
const oystehr = new Oystehr({
  accessToken: "<your_access_token>",
});
 
// Only retrieve id, name, and birthDate fields
const patientBundle = await oystehr.fhir.search<Patient>({
  resourceType: 'Patient',
  params: [
    { name: '_elements', value: 'id,name,birthDate' },
  ],
});

When working with resources that contain large arrays, using _elements can dramatically reduce response size.

3. Limit Included Resources

The _include parameter lets you retrieve related resources in one request, but it can significantly increase response size. Use it selectively and combine with _elements when needed:

import { Encounter } from 'fhir/r4b';
import Oystehr from '@oystehr/sdk';
 
const oystehr = new Oystehr({
  accessToken: "<your_access_token>",
});
 
// Only include essential fields
const encounters = await oystehr.fhir.search<Encounter>({
  resourceType: 'Encounter',
  params: [
    { name: '_include', value: 'Encounter:patient' },
    { name: '_elements', value: 'id,status,period' },
  ],
});