Virtual File System for Node.js

[mcollina]

A first-class virtual file system module (node:vfs) with a provider-based architecture that integrates with Node.js's fs module and module loader.

Key Features

• Provider Architecture - Extensible design with pluggable providers:

  • MemoryProvider - In-memory file system with full read/write support
  • SEAProvider - Read-only access to Single Executable Application assets
  • VirtualProvider - Base class for creating custom providers

• Standard fs API - Uses familiar writeFileSync, readFileSync, mkdirSync instead of custom methods

• Mount Mode - VFS mounts at a specific path prefix (e.g., /virtual), clear separation from real filesystem

• Module Loading - require() and import work seamlessly from virtual files

• SEA Integration - Assets automatically mounted at /sea when running as a Single Executable Application

• Full fs Support - readFile, stat, readdir, exists, streams, promises, glob, symlinks

Example

const vfs = require('node:vfs');
const fs = require('node:fs');

// Create a VFS with default MemoryProvider
const myVfs = vfs.create();

// Use standard fs-like API
myVfs.mkdirSync('/app');
myVfs.writeFileSync('/app/config.json', '{"debug": true}');
myVfs.writeFileSync('/app/module.js', 'module.exports = "hello"');

// Mount to make accessible via fs module
myVfs.mount('/virtual');

// Works with standard fs APIs
const config = JSON.parse(fs.readFileSync('/virtual/app/config.json', 'utf8'));
const mod = require('/virtual/app/module.js');

// Cleanup
myVfs.unmount();

SEA Usage

When running as a Single Executable Application, bundled assets are automatically available:

const fs = require('node:fs');

// Assets are automatically mounted at /sea - no setup required
const config = fs.readFileSync('/sea/config.json', 'utf8');
const template = fs.readFileSync('/sea/templates/index.html', 'utf8');

Public API

const vfs = require('node:vfs');

vfs.create([provider][, options])  // Create a VirtualFileSystem
vfs.VirtualFileSystem              // The main VFS class
vfs.VirtualProvider                // Base class for custom providers
vfs.MemoryProvider                 // In-memory provider
vfs.SEAProvider                    // SEA assets provider (read-only)

---

Disclaimer: I've used a significant amount of Claude Code tokens to create this PR. I've reviewed all changes myself.

---

Fixes #60021

[nodejs-github-bot]

Review requested:

• @nodejs/single-executable
• @nodejs/test_runner

[github-actions]

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @avivkeller.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

[Ethan-Arrowood]

Nice! This is a great addition. Since it's such a large PR, this will take me some time to review. Will try to tackle it over the next week.

[avivkeller]

Can we use path.join?

[avivkeller]

Shouldn't we import this at the top level / lazy load it at the top level?

[avivkeller]

Suggested change

	restore: restoreFS,
	restore: ctx.restore,

nit

[avivkeller]

Can we use path.join?

[avivkeller]

Let's use path.join

[avivkeller]

path.join?

[avivkeller]

Suggested change

	}).catch((err) => {
	}, (err) => {

[avivkeller]

Primordials?

[avivkeller]

Suggested change

	}).catch((err) => {
	}, (err) => {

[avivkeller]

Left an initial review, but like @Ethan-Arrowood said, it'll take time for a more in depth look

[joyeecheung]

It's nice to see some momentum in this area, though from a first glance it seems the design has largely overlooked the feedback from real world use cases collected 4 years ago: https://github.com/nodejs/single-executable/blob/main/docs/virtual-file-system-requirements.md - I think it's worth checking that the API satisfies the constraints that users of this feature have provided, to not waste the work that have been done by prior contributors to gather them, or having to reinvent it later (possibly in a breaking manner) to satisfy these requirements from real world use cases.

[codecov]

Codecov Report

❌ Patch coverage is 92.89204% with 611 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.79%. Comparing base (e0928d6) to head (8728a45).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/vfs/setup.js	83.25%	255 Missing ⚠️
lib/internal/vfs/providers/memory.js	88.63%	91 Missing ⚠️
lib/internal/vfs/providers/real.js	85.26%	56 Missing ⚠️
lib/internal/vfs/file_system.js	96.58%	46 Missing ⚠️
lib/internal/vfs/watcher.js	92.83%	35 Missing and 3 partials ⚠️
lib/internal/vfs/streams.js	89.17%	29 Missing ⚠️
lib/internal/fs/cp/cp-sync.js	56.52%	20 Missing ⚠️
src/node_sea.cc	64.28%	12 Missing and 8 partials ⚠️
lib/internal/vfs/provider.js	97.17%	12 Missing and 4 partials ⚠️
lib/internal/vfs/stats.js	93.84%	12 Missing ⚠️
... and 8 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #61478      +/-   ##
==========================================
+ Coverage   89.68%   89.79%   +0.10%     
==========================================
  Files         676      692      +16     
  Lines      206555   215088    +8533     
  Branches    39552    41196    +1644     
==========================================
+ Hits       185249   193138    +7889     
- Misses      13444    14045     +601     
- Partials     7862     7905      +43     

Files with missing lines	Coverage Δ
lib/fs.js	98.53% <100.00%> (+0.34%)	⬆️
lib/internal/bootstrap/realm.js	96.21% <100.00%> (+<0.01%)	⬆️
lib/internal/fs/utils.js	99.68% <100.00%> (+<0.01%)	⬆️
lib/internal/modules/cjs/loader.js	98.20% <100.00%> (+0.05%)	⬆️
lib/internal/modules/esm/get_format.js	94.83% <100.00%> (ø)
lib/internal/modules/esm/load.js	91.47% <100.00%> (ø)
lib/internal/modules/esm/resolve.js	99.03% <100.00%> (-0.01%)	⬇️
lib/internal/modules/esm/translators.js	97.66% <100.00%> (ø)
lib/internal/modules/helpers.js	98.73% <100.00%> (+0.01%)	⬆️
lib/internal/modules/package_json_reader.js	99.72% <100.00%> (+<0.01%)	⬆️
... and 26 more

... and 33 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jimmywarting]

And why not something like OPFS aka whatwg/fs?

const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.mount('/app', rootHandle) // to make it work with fs
fs.readFileSync('/app/config.json')

OR

const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })

fs.readFileSync('sandbox:/config.json')

fs.createVirtual seems like something like a competing specification

[mcollina]

And why not something like OPFS aka whatwg/fs?

I generally prefer not to interleave with WHATWG specs as much as possible for core functionality (e.g., SEA). In my experience, they tend to perform poorly on our codebase and remove a few degrees of flexibility. (I also don't find much fun in working on them, and I'm way less interested in contributing to that.)

On an implementation side, the core functionality of this feature will be identical (technically, it's missing writes that OPFS supports), as we would need to impact all our internal fs methods anyway.

If this lands, we can certainly iterate on a WHATWG-compatible API for this, but I would not add this to this PR.

[juliangruber]

Small prior art: https://github.com/juliangruber/subfs

[Qard]

I also worked on this a bit on the side recently: Qard@73b8fc6

That is very much in chaotic ideation stage with a bunch of LLM assistance to try some different ideas, but the broader concept I was aiming for was to have a VirtualFileSystem type which would actually implement the entire API surface of the fs module, accepting a Provider type to delegate the internals of the whole cluster of file system types to a singular class managing the entire cluster of fs-related types such that the fs module could actually just be fully converted to:

module.exports = new VirtualFileSystem(new LocalProvider())

I intended for it to be extensible for a bunch of different interesting scenarios, so there's also an S3 provider and a zip file provider there, mainly just to validate that the model can be applied to other varieties of storage systems effectively.

Keep in mind, like I said, the current state is very much just ideation in a branch I pushed up just now to share, but I think there are concepts for extensibility in there that we could consider to enable a whole ecosystem of flexible storage providers. 🙂

Personally, I would hope for something which could provide both read and write access through an abstraction with swappable backends of some variety, this way we could pass around these virtualized file systems like objects and let an ecosystem grow around accepting any generalized virtual file system for its storage backing. I think it'd be very nice for a lot of use cases like file uploads or archive management to be able to just treat them like any other readable and writable file system.

[jimmywarting]

Personally, I would hope for something which could provide both read and write access through an abstraction with swappable backends of some variety, this way we could pass around these virtualized file systems like objects and let an ecosystem grow around accepting any generalized virtual file system for its storage backing. I think it'd be very nice for a lot of use cases like file uploads or archive management to be able to just treat them like any other readable and writable file system.

just a bit off topic... but this reminds me of why i created this feature request:
Blob.from() for creating virtual Blobs with custom backing storage

Would not lie, it would be cool if NodeJS also provided some type of static Blob.from function to create virtual lazy blobs. could live on fs.blobFrom for now...

example that would only work in NodeJS (based on how it works internally)

const size = 26

const blobPart = BlobFrom({
  size,
  stream (start, end) {
    // can either be sync or async (that resolves to a ReadableStream)
    // return new Response('abcdefghijklmnopqrstuvwxyz'.slice(start, end)).body
    // return new Blob(['abcdefghijklmnopqrstuvwxyz'.slice(start, end)]).stream()
    
    return fetch('https://httpbin.dev/range/' + size, {
      headers: {
        range: `bytes=${start}-${end - 1}`
      }
    }).then(r => r.body)
  }
})

blobPart.text().then(text => {
  console.log('a-z', text)
})

blobPart.slice(-3).text().then(text => {
  console.log('x-z', text)
})

const a = blobPart.slice(0, 6)
a.text().then(text => {
  console.log('a-f', text)
})

const b = a.slice(2, 4)
b.text().then(text => {
  console.log('c-d', text)
})

x-z xyz
a-z abcdefghijklmnopqrstuvwxyz
a-f abcdef
c-d cd

An actual working PoC

(I would not rely on this unless it became officially supported by nodejs core - this is a hack)

const blob = new Blob()
const symbols = Object.getOwnPropertySymbols(blob)
const blobSymbol = symbols.map(s => [s.description, s])
const symbolMap = Object.fromEntries(blobSymbol)
const {
  kHandle,
  kLength,
} = symbolMap

function BlobFrom ({ size, stream }) {
  const blob = new Blob()
  if (size === 0) return blob

  blob[kLength] = size
  blob[kHandle] = {
    span: [0, size],

    getReader () {
      const [start, end] = this.span
      if (start === end) {
        return { pull: cb => cb(0) }
      }

      let reader

      return {
        async pull (cb) {
          reader ??= (await stream(start, end)).getReader()
          const {done, value} = await reader.read()
          cb(done ^ 1, value)
        }
      }
    },

    slice (start, end) {
      const [baseStart] = this.span

      return {
        span: [baseStart + start, baseStart + end],
        getReader: this.getReader,
        slice: this.slice,
      }
    }
  }

  return blob
}

currently problematic to do: new Blob([a, b]), new File([blobPart], 'alphabet.txt', { type: 'text/plain' })

also need to handle properly clone, serialize & deserialize, if this where to be sent of to another worker - then i would transfer a MessageChannel where the worker thread asks main frame to hand back a transferable ReadableStream when it needs to read something.

but there are probably better ways to handle this internally in core with piping data directly to and from different destinations without having to touch the js runtime? - if only getReader could return the reader directly instead of needing to read from the ReadableStream using js?

[boneskull]

@mcollina This is caldissimo.

I'm curious if you see a way to build "snapshot" functionality onto this. For example, say I wanted to save a snapshot of a directory in my VFS to a CBOR file on disk, and parse and load it into a new VFS later. Doable?