Scene zarr file specification for explore()#89
Merged
brendancol merged 11 commits intomasterfrom Mar 10, 2026
Merged
Conversation
Adds a spec document defining the on-disk zarr format for explore() scenes and a validate_scene() function to check conformance.
Covers terrain tile roughness, terrain LOD tiling parameters, and pre-simplified mesh LOD arrays. Adds validation and tests for roughness metadata and partial mesh LOD detection.
Sphere geometries now spec classification, intensity, rgb, return_number, and number_of_returns arrays for LiDAR data. Validation checks attribute length consistency against point count.
spatial_ref is the single source of truth for the scene CRS. New "Coordinate reference system" section explains how CRS applies across all groups. Wind/weather grid_bounds explicitly noted as scene CRS. Optional epsg attribute on spatial_ref for quick lookup. Point cloud source_crs is provenance only.
) build_scene(bounds, output_path) fetches 30m Copernicus DEM, Overture buildings/water, Open-Meteo wind/weather, and writes everything into a single spec-conforming scene zarr. explore_scene() opens it. Available as: - rtxpy.build_scene(bounds, "scene.zarr") - rtxpy.explore_scene("scene.zarr") - rtxpy-build-scene -112.2 36.0 -112.0 36.2 scene.zarr - python -m rtxpy.scene -112.2 36.0 -112.0 36.2 scene.zarr
Mesh arrays and wind/weather/hydro data were using plain zstd. Switching to BloscCodec(zstd, bitshuffle, clevel=6) cuts scene file size nearly in half (448 KB → 229 KB on test scene).
36 countries, 36 cities, 47 landscapes — all (west, south, east, north) in WGS 84. Includes find() for substring search and list_locations() for browsing. Wired into __init__.py as COUNTRIES, CITIES, LANDSCAPES.
Location is now a tuple subclass with a .crs attribute — unpacking as (west, south, east, north) still works, so build_scene(loc, ...) is backwards compatible. Countries use well-known national grids (EPSG:27700 for UK, EPSG:5070 for US, EPSG:2056 for Switzerland, etc). Cities and landscapes get UTM computed from the bbox center.
Location now carries a .units attribute ('meters' for all current CRS).
_CRS_UNITS lookup table makes it easy to add feet-based CRS later.
README explains why UTM for cities/landscapes and national grids for
countries — the viewer needs metric coordinates for terrain, buildings,
and particle simulations.
#88) 1. Fix wind/weather 414 URI Too Large — adaptive grid_size scales to bbox extent, capped at 12 (144 points, ~3600 chars in URL) 2. Auto-detect CRS from Location objects — build_scene(loc, ...) uses loc.crs when crs=None 3. Add roads (Overture, on by default) and fires (FIRMS, opt-in) 4. progress= callback for structured logging (replaces print()) 5. Parallel network fetches — buildings, roads, water, fires run concurrently via ThreadPoolExecutor; wind + weather also parallel 6. resume=True skips groups already present in the zarr 7. Precompute per-tile elevation roughness for LOD (bilinear corner fit residual std, written to elevation_roughness/ group) 8. CLI: --no-roads, --fires, --resume flags 8 new tests covering adaptive grid, roughness, Location, new CLI flags.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Formalizes the on-disk zarr format used by explore() into a specification document. The format already exists in pieces across remote_data.py, mesh_store.py, explore_zarr.py, and quickstart.py — this pulls them into one reference and adds groups for data that explore() accepts as in-memory arguments but doesn't persist yet (overlays, wind, hydro, weather, camera state, tours, observers).
Also adds
validate_scene()to check whether a zarr store conforms to the spec, and 21 tests covering all validation paths.The spec covers:
Closes #88
Test plan
validate_scene()returns no errors for minimal valid scene (elevation + spatial_ref)