A set of utilities that extend the use of JSON:

• parseChunked() – functions like JSON.parse() but iterates over chunks, reconstructing the result object.
• stringifyChunked() – functions like JSON.stringify(), but returns a generator yielding strings instead of a single string.
• stringifyInfo() – returns an object with the expected overall size of the stringify operation and any circular references.
• parseFromWebStream() – a helper function to consume chunks from a Web Stream.
• createStringifyWebStream() – a helper to create a Web Stream.

Features:

• Fast and memory-efficient
• Compatible with browsers, Node.js, Deno, Bun
• Supports Node.js and Web streams
• Dual package: ESM and CommonJS
• No dependencies
• Size: 9.4Kb (minified), 3.6Kb (min+gzip)

• Prevents main thread freezing during large JSON parsing by distributing the process over time.
• Handles large JSON processing (e.g., V8 has a limitation for strings ~500MB, making JSON larger than 500MB unmanageable).
• Reduces memory pressure. JSON.parse() and JSON.stringify() require the entire JSON content before processing. parseChunked() and stringifyChunked() allow processing and sending data incrementally, avoiding large memory consumption at a single time point and reducing GC pressure.

npm install @discoveryjs/json-ext

Functions like JSON.parse(), iterating over chunks to reconstruct the result object, and returns a Promise.

Note: reviver parameter is not supported yet.

function parseChunked(input: Iterable<Chunk> | AsyncIterable<Chunk>): Promise<any>;
function parseChunked(input: () => (Iterable<Chunk> | AsyncIterable<Chunk>)): Promise<any>;

type Chunk = string | Buffer | Uint8Array;

Benchmark

Usage:

import { parseChunked } from '@discoveryjs/json-ext';

const data = await parseChunked(chunkEmitter);

Parameter chunkEmitter can be an iterable or async iterable that iterates over chunks, or a function returning such a value. A chunk can be a string, Uint8Array, or Node.js Buffer.

Examples:

• Generator:

  parseChunked(function*() {
      yield '{ "hello":';
      yield Buffer.from(' "wor'); // Node.js only
      yield new TextEncoder().encode('ld" }'); // returns Uint8Array
  });

• Async generator:

  parseChunked(async function*() {
      for await (const chunk of someAsyncSource) {
          yield chunk;
      }
  });

• Array:

  parseChunked(['{ "hello":', ' "world"}'])

• Function returning iterable:

  parseChunked(() => ['{ "hello":', ' "world"}'])

• Node.js Readable stream:

  import { parseChunked } from '@discoveryjs/json-ext';
  import fs from 'node:fs';
  
  parseChunked(fs.createReadStream('path/to/file.json'))

• Web stream (e.g., using fetch()):

  Note: Iterability for Web streams was added later in the Web platform, not all environments support it. Consider using parseFromWebStream() for broader compatibility.

  const response = await fetch('https://example.com/data.json');
  const data = await parseChunked(response.body); // body is ReadableStream

Functions like JSON.stringify(), but returns a generator yielding strings instead of a single string.

Note: Returns "null" when JSON.stringify() returns undefined (since a chunk cannot be undefined).

function stringifyChunked(value: any, replacer?: Replacer, space?: Space): Generator<string, void, unknown>;
function stringifyChunked(value: any, options: StringifyOptions): Generator<string, void, unknown>;

type Replacer =
    | ((this: any, key: string, value: any) => any)
    | (string | number)[]
    | null;
type Space = string | number | null;
type StringifyOptions = {
    replacer?: Replacer;
    space?: Space;
    highWaterMark?: number;
};

Benchmark

Usage:

import { stringifyStream } from '@discoveryjs/json-ext';

const chunks = [...stringifyChunked(data)];
// or
for (const chunk of stringifyChunked(data)) {
    console.log(chunk);
}

Examples:

• Streaming into a file (Node.js):

  Readable.from(stringifyChunked(data))
      .pipe(fs.createWriteStream('path/to/file.json'));

• Wrapping into a Promise for piping into a writable Node.js stream:

  new Promise((resolve, reject) => {
      Readable.from(stringifyChunked(data))
          .on('error', reject)
          .pipe(stream)
          .on('error', reject)
          .on('finish', resolve);
  });

• Using with fetch (JSON streaming):

  Note: This feature has limited support in browsers, see Streaming requests with the fetch API

  Note: ReadableStream.from() has limited support in browsers, use createStringifyWebStream() instead.

  fetch('http://example.com', {
      method: 'POST',
      duplex: 'half',
      body: ReadableStream.from(stringifyChunked(data))
  });

• Wrapping into ReadableStream:

  Note: Use ReadableStream.from() or createStringifyWebStream() when no extra logic is needed

  new ReadableStream({
      start() {
          this.generator = stringifyChunked(data);
      },
      pull(controller) {
          const { value, done } = this.generator.next();
  
          if (done) {
              controller.close();
          } else {
              controller.enqueue(value);
          }
      },
      cancel() {
          this.generator = null;
      }
  });

export function stringifyInfo(value: any, replacer?: Replacer, space?: Space): StringifyInfoResult;
export function stringifyInfo(value: any, options?: StringifyInfoOptions): StringifyInfoResult;

type StringifyInfoOptions = {
    replacer?: Replacer;
    space?: Space;
    continueOnCircular?: boolean;
}
type StringifyInfoResult = {
    minLength: number;
    circular: Object[]; // list of circular references
};

Functions like JSON.stringify(), but returns an object with the expected overall size of the stringify operation and a list of circular references.

Example:

import { stringifyInfo } from '@discoveryjs/json-ext';

console.log(stringifyInfo({ test: true }));
// {
//   bytes: 13, // Buffer.byteLength('{"test":true}')
//   circular: []    
// }

Type: Boolean
Default: false

Determines whether to continue collecting info for a value when a circular reference is found. Setting this option to true allows finding all circular references.

A helper function to consume JSON from a Web Stream. You can use parseChunked(stream) instead, but @@asyncIterator on ReadableStream has limited support in browsers (see ReadableStream compatibility table).

import { parseFromWebStream } from '@discoveryjs/json-ext';

const data = await parseFromWebStream(readableStream);
// equivalent to (when ReadableStream[@@asyncIterator] is supported):
// await parseChunked(readableStream);

A helper function to convert stringifyChunked() into a ReadableStream (Web Stream). You can use ReadableStream.from() instead, but this method has limited support in browsers (see ReadableStream.from() compatibility table).

import { createStringifyWebStream } from '@discoveryjs/json-ext';

createStringifyWebStream({ test: true });
// equivalent to (when ReadableStream.from() is supported):
// ReadableStream.from(stringifyChunked({ test: true }))

MIT