feat: Add RocksDB snapshot download RPC API with streaming support

[gshep]

Resolves #5177

• Introduce snapshot RPC API with manifest, chunk, and completion messages
• Implement snapshot service creating RocksDB checkpoints and zstd-compressed tar archives
• Limit concurrent downloads and clean up stale snapshots automatically
• Add CLI example for snapshot download verification
• Add config flags for enabling snapshot RPC and related parameters

@gear-tech/dev

[semanticdiff-com]

Review changes with  

Changed Files

File	Status
ethexe/cli/src/params/mod.rs	60% smaller
ethexe/db/src/rocks.rs	22% smaller
ethexe/rpc/src/lib.rs	13% smaller
ethexe/cli/src/params/rpc.rs	2% smaller
ethexe/rpc/src/tests.rs	2% smaller
.ethexe.example.local.toml	Unsupported file format
.ethexe.example.toml	Unsupported file format
Cargo.lock	Unsupported file format
ethexe/rpc/Cargo.toml	Unsupported file format
ethexe/rpc/examples/snapshot_download_verify.rs	0% smaller
ethexe/rpc/src/apis/mod.rs	0% smaller
ethexe/rpc/src/apis/snapshot.rs	0% smaller
ethexe/rpc/src/errors.rs	0% smaller
ethexe/service/src/lib.rs	Unsupported file format
ethexe/service/src/tests/mod.rs	0% smaller
ethexe/service/src/tests/utils/env.rs	0% smaller
utils/gear-workspace-hack/Cargo.toml	Unsupported file format

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the node's capabilities by introducing a robust and secure RPC API for downloading RocksDB snapshots. This feature allows clients to efficiently retrieve a consistent state of the database, which is crucial for node synchronization, recovery, or data analysis. The implementation includes streaming support for large datasets, configurable parameters for performance and resource management, and built-in authentication to protect access to sensitive snapshot data.

Highlights

• RocksDB Snapshot RPC API: Introduced a new RPC API for downloading RocksDB snapshots, supporting streaming of manifest, chunk, and completion messages.
• Snapshot Service Implementation: Implemented a dedicated snapshot service responsible for creating RocksDB checkpoints, compressing them into zstd-compressed tar archives, and managing their lifecycle.
• Concurrency Control and Cleanup: Added mechanisms to limit concurrent snapshot downloads and automatically clean up stale snapshot artifacts to manage resource usage.
• Configuration and CLI Support: Integrated new configuration flags for enabling the snapshot RPC and setting related parameters (e.g., token, chunk size, retention, concurrency). A CLI example for snapshot download verification was also added.
• Authentication Middleware: Implemented an RPC middleware for snapshot methods to enforce bearer token authentication, enhancing security for snapshot downloads.

Changelog

• .ethexe.example.local.toml
  • Added new configuration options for RocksDB snapshot RPC, including snapshot, snapshot-token, snapshot-chunk-bytes, snapshot-retention-secs, and snapshot-max-concurrent.
• .ethexe.example.toml
  • Added new configuration options for RocksDB snapshot RPC, including snapshot, snapshot-token, snapshot-chunk-bytes, snapshot-retention-secs, and snapshot-max-concurrent.
• Cargo.lock
  • Updated dependencies to include clap 4.5.54, hex, sha2 0.10.9, tar, tempfile, zstd 0.13.3, and zstd-sys.
• ethexe/cli/src/params/mod.rs
  • Modified RPC parameter processing to correctly handle the new snapshot configuration, updating the into_config method to return a Result<Option<RpcConfig>>.
• ethexe/cli/src/params/rpc.rs
  • Extended RpcParams with new fields for snapshot configuration (e.g., snapshot, snapshot_token, snapshot_chunk_bytes), updated into_config to validate and construct SnapshotRpcConfig, and added tests for empty snapshot tokens and gas allowance overflow.
• ethexe/db/src/rocks.rs
  • Added a create_checkpoint method to RocksDatabase for generating physical RocksDB checkpoints and included a test case for checkpoint creation and reopening.
• ethexe/rpc/Cargo.toml
  • Added a new example target snapshot_download_verify and introduced new dependencies sha2, hex, tar, zstd, clap, tempfile, and tokio for dev-dependencies.
• ethexe/rpc/examples/snapshot_download_verify.rs
  • Added a new CLI example for verifying RocksDB snapshot downloads, including functionality to spawn a local RPC server, download snapshots, and verify their integrity and content.
• ethexe/rpc/src/apis/mod.rs
  • Exported new modules and types related to the snapshot RPC API, including snapshot module, SnapshotApi, SnapshotServer, and SnapshotStreamItem.
• ethexe/rpc/src/apis/snapshot.rs
  • Introduced a new snapshot API module defining the Snapshot RPC trait, SnapshotApi implementation, and SnapshotService for managing snapshot creation, archiving, streaming, and cleanup processes.
• ethexe/rpc/src/errors.rs
  • Added new error helper functions: internal_with, unauthorized, and unavailable for more specific error reporting in RPC responses.
• ethexe/rpc/src/lib.rs
  • Integrated SnapshotStreamItem and SnapshotClient exports, introduced SnapshotRpcConfig, updated RpcServer to accept an optional RocksDatabase for snapshot source, and implemented SnapshotAuthRpcMiddleware for bearer token authentication on snapshot methods.
• ethexe/rpc/src/tests.rs
  • Added comprehensive unit and integration tests for the new snapshot RPC API, covering successful downloads, authentication failures, method registration, streaming behavior, cleanup logic, and concurrency limits.
• ethexe/service/src/lib.rs
  • Modified the Service initialization to pass the RocksDatabase instance to the RpcServer when a snapshot RPC is configured, enabling the snapshot service to access the database.
• ethexe/service/src/tests/mod.rs
  • Updated RPC configuration in tests to explicitly set snapshot: None to avoid unintended snapshot RPC activation.
• ethexe/service/src/tests/utils/env.rs
  • Updated default RPC configuration in test utilities to explicitly set snapshot: None.
• utils/gear-workspace-hack/Cargo.toml
  • Added zstd-sys as a dependency to the workspace hack.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a valuable feature for downloading RocksDB snapshots via a streaming RPC API. While the implementation is generally well-structured, includes a detailed CLI example, and handles authentication and concurrency control correctly, a medium-severity security vulnerability has been identified: sensitive snapshots are insecurely stored in the system's temporary directory, which could lead to local information disclosure of the entire node database. Furthermore, a critical issue in the cleanup logic and a potential consistency issue regarding the hashing algorithm used require attention.