Skip to content

Navigation Menu

Sign in
Sign up

Multi shard consolidation - #862

Open
blasscoc wants to merge 5 commits into
TGSAI:main from
blasscoc:multi-shard-consolidation
Open

Multi shard consolidation #862
blasscoc wants to merge 5 commits into
TGSAI:main from
blasscoc:multi-shard-consolidation

Conversation

@blasscoc

@blasscoc blasscoc commented Aug 6, 2026

Copy link
Copy Markdown

This is a little unhinged, but I want to post here in case there was an interest.

The multi-SEGY consolidation could also be done using Xarray directly and the "concat" feature.

Otherwise the "gun" needs to be optional.

Zarr V3 sharding on ingestion is also a win.

blasscoc-tgs and others added 5 commits July 13, 2026 02:01
Additive feature for consolidating many SEG-Y shards (e.g. a shot survey split
across files) into one MDIO store, cloud-to-cloud, without changing the default
single-file segy_to_mdio behavior.
- allocate_mdio_grid / append_segy_shard: two-phase region-write workflow that
 builds an empty global grid, then writes each shard into its region in place.
- plan_consolidation (+ ConsolidationPlan): deterministic planner computing per
 shard chunk ownership, per-chunk fill vs merge (read-modify-write) mode, the
 shard conflict graph, and concurrency waves. Orchestration is left to callers.
- Optional read-modify-write in trace_worker / blocked_io.to_zarr via a
 merge_chunks set, so shards sharing a boundary chunk don't clobber each other.
 Defaults to the original fast pure-write path when not provided.
- Export the new API from mdio, mdio.converters.
A shared template's "required" set should describe the data category, not one
contributor's acquisition rig. `gun` (StreamerShotGathers3D) and the new
elevation coords (ShotReceiverLineGathers3D) are now optional coordinates:
populated when the source carries them, silently omitted otherwise.
- base: add _optional_coord_names + optional_coordinate_names property
- validation: subtract optional coords from required-field check; add
 prune_absent_optional_coordinates to drop them from the resolved schema
 when the segy_spec doesn't carry them
- pipeline + consolidate: prune optional coords after schema resolution
- shot-receiver-line: add optional source/receiver elevation for true 3D
 land/areal geometry
Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce an optional shard grid so many chunks pack into one storage object
(far fewer objects / S3 GETs) while keeping the small chunk as the partial-read
unit. Adds shard_grid to VariableMetadata and shard_shape to ResolvedSchema,
a full_shard_shape property on the template base, resolver pass-through, and
shard resolution/validation (whole multiple of the chunk per dimension) in the
dataset factory. The xarray builder emits the sharding_indexed codec (v3 only)
and aligns Dask lazy blocks to shard boundaries so to_zarr's safe-chunks guard
passes; blocked_io writes whole shards to avoid read-modify-write races.
Headers are intentionally left unsharded: their structured (void) dtype trips
Zarr's sharding partial-encode path, and they are tiny relative to the sample
cube. Sharding applies to the data variable.
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /