uhopuhop
907

Stream Json

screenshot of Stream Json
django

The micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API.

Github

Overview:

stream-json is a micro-library of node.js stream components that allows for processing huge JSON files with minimal memory requirements. It offers a range of features, including a streaming JSON parser, filters for editing token streams, streamers for producing streams of JavaScript objects, and utilities for emitting tokens as events, batch processing, verifying JSON validity, and handling JSONL (JSON Lines) files. The components can be used together with stream-chain to create custom data processing pipelines.

Features:

  • Streaming JSON Parser: Parses JSON files and produces a SAX-like token stream.
  • Packable Tokens: The parser can optionally pack keys, strings, and numbers separately.
  • Filtering: Provides filters to edit the token stream, allowing for selecting desired objects, replacing objects, and removing objects.
  • Streamers: Includes streamers for producing a stream of JavaScript objects, such as StreamValues for handling selected objects and StreamArray for streaming array components.
  • Essential Components: Offers an assembler for interpreting token streams and creating JavaScript objects, a disassembler for producing token streams from JavaScript objects, a stringer for converting token streams into a JSON text stream, and an emitter for reading token streams and emitting tokens as events.
  • Utilities: Provides additional utilities for batch processing, verifying JSON validity, sanitizing multibyte UTF8 text input, and more.
  • JSONL Support: Includes components specifically designed for parsing and producing JSONL files (also known as JSON Lines or NDJSON).

Summary:

stream-json is a powerful micro-library for processing huge JSON files with minimal memory requirements. It offers a range of components including a streaming JSON parser, filters for editing token streams, streamers for producing JavaScript object streams and utilities for emitting tokens as events, batch processing, and more. The library also provides support for handling JSONL files. With stream-json, you can easily create custom data processing pipelines to handle large JSON datasets efficiently.

django
Django

Django is a high-level Python web framework that encourages rapid development and clean, pragmatic design. It follows the model-view-controller (MVC) architectural pattern, providing an extensive set of built-in tools and conventions to streamline the creation of robust and scalable web applications.