Package 'blocklist'

Build a sources table for virtualize_mosaic()

Description

Pairs each source file's access path (what ⁠gdal mdim mosaic⁠ is given and what rhdf5 opens to scan) with its public URL (what is written into the reference store). Chunk byte offsets are identical in any byte-identical copy of a file, so the two may differ freely: scan a cheap local or NFS path while referencing a durable remote URL.

Usage

mosaic_sources(public, access = public)

Arguments

public	Character vector of durable URLs to record in the references (e.g. ⁠https://⁠ or ⁠s3://⁠), one per source file.
access	Character vector, same order as public, of the paths actually opened during the build (handed to ⁠gdal mdim mosaic⁠ and opened by rhdf5). Defaults to public for the all-remote case.

Value

A data.frame with columns access and public, one row per source.

See Also

virtualize_mosaic()

Examples

## Not run: 
mosaic_sources(
  public = c("https://host/day01.nc", "https://host/day02.nc"),
  access = c("/nfs/day01.nc",        "/nfs/day02.nc"))

## End(Not run)

---

Parse a GDAL multidimensional mosaic VRT

Description

Reads the logical manifest produced by ⁠gdal mdim mosaic⁠: dimension sizes, and for each data array its C-order dimension names, dtype, shape, GDAL block size, CF scale/offset/nodata/unit, and the ordered list of sources (each with SourceFilename, SourceArray, and DestSlab offset). Coordinate arrays are returned with their composed values (from the VRT's InlineValues or RegularlySpacedValues). No data or chunk bytes are read.

Usage

parse_mosaic_vrt(vrt)

Arguments

vrt	Path to a .vrt written by ⁠gdal mdim mosaic⁠.

Value

A list with dim_size (named integer vector of dimension extents), arrays (named list of data-variable descriptions, each with dim_names, dtype, shape, chunks, scale, offset, nodata, unit, sources), and coords (named list with dim, dtype, values).

See Also

virtualize_mosaic()

---

Extract chunk byte references for one array in one source file

Description

Opens source_array in scan_path read-only and walks its HDF5 chunk index (h5getAllChunkInfo) to recover, per chunk, the grid coordinate (C order) and the byte address and length of the stored, compressed data. Coordinates are local to the source; the caller shifts them to global position using the mosaic DestSlab offset. A non-zero filter mask on any chunk aborts with an error - it means the chunk does not share the array-level codec and cannot be virtualised uniformly.

Usage

scan_source_chunks(scan_path, source_array, ref_path, A, contiguous)

Arguments

scan_path	Path opened to read the chunk index (the source's access).
source_array	Array name within the file, e.g. "/anom".
ref_path	Durable URL written into the path column (the public URL).
A	Array description from parse_mosaic_vrt() with at least shape and chunks (C order); chunks must be the authoritative HDF5 storage chunk.
contiguous	Logical; if TRUE the dataset is unchunked and a single whole-array reference is emitted instead of iterating a chunk index.

Value

A data.frame, one row per chunk: ndim integer coordinate columns (c1..cN), offset (byte address), size (byte length), path.

---

Virtualise a GDAL mosaic into a kerchunk-Parquet reference store

Description

End-to-end driver. Parses a ⁠gdal mdim mosaic⁠ VRT for the logical layout, probes each array's codec and storage chunk shape once via rhdf5, scans every source file for chunk byte references, composes them across files using the VRT DestSlab offsets, inlines coordinate variables (values from the VRT, CF attributes read from a source), and writes the store. The output is both a portable reference store and the input to a VirtualiZarr -> Icechunk step.

Usage

virtualize_mosaic(vrt, root, sources, record_size = 100000L)

Arguments

vrt	Path to a VRT written by ⁠gdal mdim mosaic⁠ over sources$access.
root	Output directory for the store.
sources	A data.frame with columns access and public, typically from mosaic_sources(). Joined to the VRT by access (the SourceFilename), with a basename fallback.
record_size	Integer chunk references per parquet shard (default 1e5).

Value

The store root, invisibly.

See Also

mosaic_sources(), parse_mosaic_vrt(), write_kerchunk_parquet()

Examples

## Not run: 
src <- mosaic_sources(public = urls, access = nfs_paths)
system(sprintf("gdal mdim mosaic %s mosaic.vrt", paste(src$access, collapse = " ")))
virtualize_mosaic("mosaic.vrt", "oisst.zarr", sources = src)

## End(Not run)

---

Writes the fsspec LazyReferenceMapper layout (Zarr v2): a root .zmetadata holding consolidated array/group metadata plus record_size, and a ⁠<var>/refs.<N>.parq⁠ shard per array. Each row is either a chunk reference (path/offset/size) or inline data (raw); rows are placed at the C-order flat chunk index and padded to record_size. The result is readable by VirtualiZarr (KerchunkParquetParser), the GDAL Zarr driver, and any fsspec reference filesystem.

Description

Writes the fsspec LazyReferenceMapper layout (Zarr v2): a root .zmetadata holding consolidated array/group metadata plus record_size, and a ⁠<var>/refs.<N>.parq⁠ shard per array. Each row is either a chunk reference (path/offset/size) or inline data (raw); rows are placed at the C-order flat chunk index and padded to record_size. The result is readable by VirtualiZarr (KerchunkParquetParser), the GDAL Zarr driver, and any fsspec reference filesystem.

Usage

write_kerchunk_parquet(
  root,
  vars_meta,
  ref_tables,
  inline = list(),
  record_size = 100000L,
  root_attrs = list()
)

Arguments

root	Output directory for the store (created, overwriting any existing).
vars_meta	Named list of per-variable metadata; each entry has shape, chunks, dtype, compressor, filters, fill_value, dim_names, attrs.
ref_tables	Named list (matching vars_meta) of chunk-reference data.frames from scan_source_chunks(); omit for variables given via inline.
inline	Named list of variables stored inline; each entry has raw, the bytes of a single whole-array chunk.
record_size	Integer chunk references per parquet shard (default 1e5).
root_attrs	Named list of group-level attributes for the root .zattrs.

Value

The store root, invisibly.

See Also

virtualize_mosaic()

---