<datastream>

Commonly used stream patterns for Web Streams API and NodeJS Stream.

If you're iterating over an array more than once, it's time to use streams.

Warning: This library is in Alpha, and will contain breaking changes as modules mature to have consistent usage patterns.

Roadmap

• Documentation

• Extend Modules

  • core
    • merge - connect multiple readable into one stream (see fetch)
  • csv
    • break into parts
      • csvDetectDelimitors({chunkSize}) then pass into csvParse as option
      • csvDetectHeaders({chunkSize}) then pass into csvArrayToObject
      • csvParse w/ hooks for error checks
      • csvRemoveMissizedRows - remove or error when note exact columns
      • csvRemoveEmptyRows - remove or error blank rows
      • csvArrayToObject -> objectFromEntriesStream
      • csvCoerceValues -> validate w/ coerce or auto cast?

• New Modules

  • json
    • parse - https://github.com/jimhigson/oboe.js / https://github.com/dscape/clarinet
    • parseChunk (json-stream notation) - fastify/secure-json-parse
    • format
    • formatChunk (json-stream notation) - fastify/fast-json-stringify
    • https://github.com/uhop/stream-json
    • https://github.com/creationix/jsonparse
    • https://github.com/dominictarr/JSONStream
    • https://www.npmjs.com/package/json-stream - string chunk -> look for \n -> JSON.parse

• Maybe Future

  • compression

    • protobuf - protobufjs / pbf - https://buf.build/blog/protobuf-es-the-protocol-buffers-typescript-javascript-runtime-we-all-deserve

  • encryption

    • encrypt [transform]
    • decrypt [transform]

  • ipfs

    • readable - get
    • writable - add https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#ipfsadddata-options https://github.com/ipfs/js-datastore-s3/blob/master/examples/full-s3-repo/index.js // https://github.com/ipfs-examples/js-ipfs-examples/blob/master/examples/browser-add-readable-stream/src/index.js

  • xml

    • parse - https://github.com/isaacs/sax-js / KeeeX/sax-js
    • format

  • type

    • pg - obj to db types

  • benchmark using node file.js | pv > /dev/null