Package 'zaro'

Open a Zarr store

Description

Connects to a Zarr store at the given path or URI. Dispatches to the appropriate backend based on the URI scheme:

• Local paths: base R (no arrow or GDAL needed)

• s3://, gs://: listed and read through Arrow when arrow is installed, else through GDAL's /vsis3/ or /vsigs/ (gdalraster). When access is known (anonymous, or S3 keys in the environment) chunks are fetched as HTTPS through the curl pool instead, see zaro_fetch()

• http://, https://: curl (no GDAL); chunk reads are batched through one connection pool, see zaro_fetch(). An HTTP store can't be listed, so it needs consolidated metadata (or array names, see zaro_dataset())

• /vsi...: any GDAL virtual file system path, through gdalraster (e.g. /vsicurl/https://... to keep GDAL's HTTP handling)

• reference+json://: Kerchunk JSON reference store

• reference+parquet://: Kerchunk Parquet reference store

• virtualizarr://: VirtualiZarr Parquet manifest store

• byterefs:// (or the older mdimrefs://): a byterefs SQLite byte-reference table (https://github.com/byterefs/spec), byterefs/0.1 or the legacy mdim-refs/0.2. Source paths are remapped for the store's meta.default_target; pass target = "public" for another target, or target = "" for the verbatim sources

• flatrefs://: a byterefs flat refs store, the Parquet export (a directory, local or remote, with arrays.json and one refs_<array>.parquet per array, as raad.virtualize publishes). Chunk references are read by row group, only those whose dim statistics can hold the selection, so a table of tens of millions of rows is never read whole. target as for byterefs://

Usage

zaro(source, verbose = zaro_verbose(), ..., validate = FALSE)

Arguments

source	character. Path, URI, or reference store locator.
verbose	logical. Emit progress and diagnostic messages. Default getOption("zaro.verbose", TRUE); set options(zaro.verbose = FALSE) to silence zaro everywhere.
...	store options: for s3:// and gs://, arguments for the Arrow filesystem constructors (e.g. anonymous = TRUE, region, endpoint_override), also used for curl-pool access; for byterefs:// and flatrefs://, target (the path remap to apply).
validate	logical. Read the root metadata at open, warning if that fails. Default FALSE.

Details

For S3 stores, if Arrow returns a 301 redirect (wrong region), zaro will fall through to gdalraster's /vsis3/ which handles region detection automatically.

Known public buckets (e.g. cmip6, aodn-cloud-optimised) are accessed anonymously by default. For other public stores, pass anonymous = TRUE.

Opening reads nothing from the store by default (source is only checked to be a non-empty character string); set validate = TRUE to read the root metadata at open.

Value

A store object (internal class) for use with other zaro functions.

Examples

## Not run: 
# local store
store <- zaro("/data/my_dataset.zarr")

# S3 store (anonymous)
store <- zaro("s3://mur-sst/zarr-v1", anonymous = TRUE)

# GCS - known public bucket, anonymous auto-detected
store <- zaro(paste0("gs://cmip6/CMIP6/ScenarioMIP/NOAA-GFDL/GFDL-ESM4/",
                     "ssp585/r1i1p1f1/Omon/zos/gn/v20180701"))

# VirtualiZarr Parquet reference
store <- zaro("virtualizarr://https://example.com/dataset.parq")

# byterefs: a SQLite table, or the flat Parquet export
store <- zaro("byterefs:///path/to/store.sqlite")
store <- zaro("flatrefs://https://example.com/store", target = "public")

# then explore
zaro_list(store)
zaro_meta(store, "analysed_sst")
data <- zaro_read(store, "analysed_sst",
                  start = c(0, 0, 0), count = c(1, 100, 100))

## End(Not run)

---

Open a Zarr array as a lazy R array

Description

Returns an ordinary R array (a double, integer or logical vector with a dim attribute and no class) whose values are read from the store chunk by chunk, only when R asks for them. It is an ALTREP object built by altarr::altarr(), with zaro's chunk fetch and codec pipeline behind it. Nothing is read when the array is created beyond the metadata.

Usage

zaro_array(
  store,
  path = "",
  meta = NULL,
  parallel = FALSE,
  verbose = zaro_verbose(),
  cf = FALSE
)

Arguments

store	a store object from zaro(), or a zaro_dataset() (whose metadata is then used, no request made).
path	character. Path to the array within the store.
meta	optional pre-fetched ZaroMeta, or the root group meta with consolidated entries, as for zaro_read().
parallel	logical. If TRUE, each batch of 4 or more chunks is fetched and decoded with future.apply::future_lapply(), as for zaro_read() (the same caveats about store serialization apply).
verbose	logical. Emit diagnostic messages when reading metadata. Default getOption("zaro.verbose", TRUE).
cf	logical. Apply CF decoding (masking and unpacking, see below). Default FALSE.

Details

Subsetting with x[i], x[cbind(i, j, k)] and altarr::altarr_extract(x, i, j, k) plans the chunks it needs and fetches them in one batch; sum(), mean(), min() and max() stream the chunk grid. See ?altarr::altarr_contract for what base R does with a lazy array, and altarr::altarr_stats() to count fetches.

Value

A lazy array, see altarr::altarr().

Dimension order

A C-order Zarr array of shape ⁠(s1, ..., sn)⁠ (all Zarr V3 arrays, and V2 arrays with "order": "C") is returned with R dim c(sn, ..., s1), and the dimension names reversed to match. This is the order the bytes are stored in, so a chunk is used as decoded. For a typical ⁠(time, lat, lon)⁠ array, x[lon, lat, time] is the R subscript. V2 arrays with "order": "F" keep their shape. This is the transpose of what zaro_read() returns.

Missing chunks and types

Chunks absent from the store read as the array's fill_value. Integer types up to 32 bits give an integer array (uint32 and 64-bit integers give double, as in zaro_read()), floats give double, bool gives logical. No CF packing (scale_factor, add_offset, ⁠_FillValue⁠) is applied unless cf = TRUE. String arrays are refused (a lazy array is numeric or logical); read them with zaro_read().

CF decoding

With cf = TRUE, values equal to the array's fill_value or to the ⁠_FillValue⁠ or missing_value attributes (and NaN) read as NA, then scale_factor and add_offset are applied. Missing chunks read as NA. A packed array reads as double; an unpacked one keeps its type. This is what xarray's default decoding and GDAL's nodata plus unscaling give.

Saving

saveRDS() on the result writes the recipe (shape, chunk shape and the fetch function), not the values. Stores backed by Arrow or GDAL hold external pointers, so a saved array does not read in a new R session; re-open it with zaro_array().

Examples

## Not run: 
store <- zaro("https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr")
x <- zaro_array(store, "precip")
dim(x)                       # longitude, latitude, time
x[cbind(180, 90, 1:10)]      # one pixel, ten time steps, one fetch
altarr::altarr_extract(x, 1:20, 1:20, 100)
altarr::altarr_stats(x)

## End(Not run)

---

Read a single chunk by grid index

Description

Fetches and decodes one chunk, identified by its position in the chunk grid. Returns the decoded values with metadata about where the chunk sits in the array.

Usage

zaro_chunk(
  store,
  path,
  chunk_idx,
  meta = NULL,
  shaped = TRUE,
  verbose = zaro_verbose()
)

Arguments

store	a store object from zaro()
path	character. Path to the array.
chunk_idx	integer vector. 0-based chunk grid index (e.g. c(0, 2, 3) for the first time step, third lat chunk, fourth lon chunk).
meta	optional pre-fetched ZaroMeta (or root group meta).
shaped	logical. If TRUE (default), reshape the values to the chunk's actual dimensions (handling C-order and edge chunks). If FALSE, return a flat vector.
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).

Value

A list with components:

cidx

integer vector - the chunk grid index

values

array or vector - decoded chunk data

start

integer vector - 0-based array coordinates of chunk origin

shape

integer vector - actual shape of this chunk (may be smaller than chunk_shape for edge chunks)

Returns NULL if the chunk doesn't exist (missing/sparse).

---

Apply a function over chunks

Description

Iterates over chunks and applies a function to each, accumulating results. Chunks are fetched and decoded one at a time (or in parallel) so memory usage is proportional to one chunk, not the full array.

Usage

zaro_chunk_apply(
  store,
  path,
  fun,
  ranges = NULL,
  meta = NULL,
  shaped = TRUE,
  parallel = FALSE,
  verbose = zaro_verbose()
)

Arguments

store	a store object from zaro()
path	character. Path to the array.
fun	function. Called with a single chunk result list (as from zaro_chunk()). NULL chunks (missing) are skipped.
ranges	optional range specification (as in zaro_chunks()).
meta	optional pre-fetched ZaroMeta (or root group meta).
shaped	logical. If TRUE, reshape each chunk's values.
parallel	logical. Use future.apply for parallel execution.
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).

Value

A list of results from fun, one per non-NULL chunk.

Examples

## Not run: 
# per-chunk mean, memory-efficient
means <- zaro_chunk_apply(store, "temperature", function(chunk) {
  mean(chunk$values, na.rm = TRUE)
})

# first time step only, all spatial chunks
means <- zaro_chunk_apply(store, "temperature", function(chunk) {
  c(mean = mean(chunk$values, na.rm = TRUE),
    n_valid = sum(is.finite(chunk$values)))
}, ranges = list(0, NULL, NULL))

## End(Not run)

---

Get chunk grid information for an array

Description

Returns the chunk grid dimensions and related metadata needed for chunk-indexed operations.

Usage

zaro_chunk_info(store, path, meta = NULL, verbose = zaro_verbose())

Arguments

store	a store object from zaro()
path	character. Path to the array.
meta	optional pre-fetched ZaroMeta (or root group meta).
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).

Value

A list with components:

grid

integer vector - number of chunks along each dimension

chunk_shape

integer vector - shape of a full (non-edge) chunk

array_shape

integer vector - shape of the full array

n_chunks

integer - total number of chunks

dimension_names

character vector or NULL

meta

ZaroMeta for the array

---

Read multiple chunks by index range

Description

Fetches and decodes a set of chunks identified by ranges along each dimension of the chunk grid, one at a time or in parallel. For batched, pooled reads use zaro_plan() and zaro_fetch(), or zaro_read().

Usage

zaro_chunks(
  store,
  path,
  ranges = NULL,
  meta = NULL,
  shaped = TRUE,
  parallel = FALSE,
  verbose = zaro_verbose()
)

Arguments

store	a store object from zaro()
path	character. Path to the array.
ranges	a list of integer vectors, one per dimension, giving the 0-based chunk indices to read. Use NULL for "all chunks along this dimension". E.g. list(0, NULL, 2:4) reads chunk 0 along dim 1, all chunks along dim 2, chunks 2-4 along dim 3.
meta	optional pre-fetched ZaroMeta (or root group meta).
shaped	logical. If TRUE, reshape each chunk's values.
parallel	logical. Use future.apply for parallel fetch+decode.
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).

Value

A list of chunk results (as from zaro_chunk()). NULL entries indicate missing chunks.

---

Open a store as a dataset: every array, its dimensions, no data read

Description

Collects the metadata of every array in one group of a Zarr store (the root by default) and reports each array's dimension names and sizes in R order (the order zaro_array() returns). Consolidated metadata (.zmetadata or a consolidated zarr.json) is used when present, which is one request for the whole store. Otherwise the store is listed and each child is probed for array metadata; pass arrays to skip the listing (useful over plain HTTP, which can't list).

Usage

zaro_dataset(source, group = "", arrays = NULL, verbose = zaro_verbose(), ...)

Arguments

source	a store from zaro(), or a character source passed to zaro() along with ....
group	path of the group to open, e.g. "measurements/reflectance/r10m". Default "", the root.
arrays	optional character vector of array names (within group) to include. With consolidated metadata it filters; without, it replaces the listing.
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).
...	passed to zaro() when source is character.

Details

No array data is read. Use zaro_array() on the result to get a lazy array, or zaro_read() with meta = ds$arrays[[name]].

Arrays with no dimension names (Zarr V2 without xarray's ⁠_ARRAY_DIMENSIONS⁠) get names "<array>_dim<k>", with k the 0-based Zarr axis, so they never collide with another array's dimensions. Only arrays directly in group are included; use zaro_groups() to see a nested store's groups (e.g. EOPF and GeoZarr products, which keep their arrays several groups down).

Value

A zaro_dataset: a list with store, group (the group's path), attributes (the group's attributes), arrays (named list of array metadata), paths (named character vector: each array's path in the store), dims (named list of R-order dimension names per array) and shape (named list of R-order sizes per array). Arrays are named within the group; zaro_array(), zaro_read() and zaro_plan() take the dataset with those names.

Examples

## Not run: 
ds <- zaro_dataset("s3://aodn-cloud-optimised/model_sea_level_anomaly_gridded_realtime.zarr",
                   anonymous = TRUE, region = "ap-southeast-2")
ds
ds$dims$GSLA                       # "LONGITUDE" "LATITUDE" "TIME"
gsla <- zaro_array(ds, "GSLA", cf = TRUE)

## a nested store: list the groups, open one
eopf_url <- paste0(
  "https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/",
  "tests-output/sentinel-2-l2a/",
  "S2C_MSIL2A_20260120T084301_N0511_R064_T36UXA_20260120T122910.zarr")
eopf <- zaro(eopf_url)
zaro_groups(eopf)
r10m <- zaro_dataset(eopf, group = "measurements/reflectance/r10m")
b04 <- zaro_array(r10m, "b04")

## End(Not run)

---

Fetch the bytes of a plan

Description

Executes a zaro_plan(): returns the raw (still encoded) bytes of every chunk, in plan order, NULL for chunks that are absent (they read as fill_value). HTTP references are fetched concurrently through one curl connection pool with a per-host cap, adjacent byte ranges are merged into single requests, and failures are retried. S3 and GCS objects join the same pool as HTTPS when their access is known: anonymous (zaro(..., anonymous = TRUE), a known public bucket, AWS_NO_SIGN_REQUEST=YES or options(zaro.cloud_anonymous = TRUE)), or S3 signed with AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (and AWS_SESSION_TOKEN) from the environment. Other credentials (profiles, instance roles, signed GCS) are read through Arrow or GDAL one object at a time. zaro_read() and zaro_array() read through this, so they get the same batching.

Usage

zaro_fetch(
  store,
  plan,
  max_con = getOption("zaro.max_con", 8L),
  merge_gap = getOption("zaro.merge_gap", 0),
  retries = 2L,
  verbose = zaro_verbose()
)

Arguments

store	the store the plan was made from.
plan	a zaro_plan (or any refs table with the same columns).
max_con	integer. Maximum simultaneous HTTP connections, per host and in total. Default getOption("zaro.max_con", 8L).
merge_gap	numeric. Merge byte ranges on the same URL separated by at most this many bytes (the gap is fetched and discarded). Default getOption("zaro.merge_gap", 0): merge only touching ranges.
retries	integer. Retries for failed HTTP requests.
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).

Value

A list of raw vectors (or NULL), one per plan row, with attribute "stats": HTTP requests, bytes received and seconds.

Examples

## Not run: 
p <- zaro_plan(store, "temp", start = c(0, 0, 0, 0), count = c(1, 1, NA, NA))
raw <- zaro_fetch(store, p)
attr(raw, "stats")

## End(Not run)

---

The groups of a store

Description

Lists every group of a Zarr store, the root included, with the number of arrays directly in each, so a nested product can be opened one group at a time with zaro_dataset(group = ). Uses consolidated metadata when the root has it (one request); otherwise lists the store, which needs a store that can be listed (not plain HTTP).

Usage

zaro_groups(source, verbose = zaro_verbose(), ...)

Arguments

source	a store from zaro(), or a character source passed to zaro() along with ....
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).
...	passed to zaro() when source is character.

Value

A data frame with columns group (path, "" for the root) and arrays (number of arrays directly in the group), sorted by path.

Examples

## Not run: 
eopf_url <- paste0(
  "https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/",
  "tests-output/sentinel-2-l2a/",
  "S2C_MSIL2A_20260120T084301_N0511_R064_T36UXA_20260120T122910.zarr")
zaro_groups(eopf_url)

## End(Not run)

---

List contents of a Zarr store

Description

List contents of a Zarr store

Usage

zaro_list(store, path = "", ...)

Arguments

store	a store object from zaro()
path	character. Path prefix within the store (default root).
...	arguments for methods (specifically 'recursive = FALSE' is default for local and Arrow stores)

Value

character vector of keys (relative paths).

---

Get metadata for a Zarr node

Description

Reads and parses the metadata for an array or group at the given path. Automatically detects Zarr V2 (.zarray/.zattrs/.zmetadata) vs V3 (zarr.json) format. For cloud stores, consolidated metadata (.zmetadata for V2, consolidated zarr.json for V3) is preferred to minimise the number of requests.

Usage

zaro_meta(store, path = "", consolidated = TRUE, verbose = zaro_verbose())

Arguments

store	a store object from zaro()
path	character. Path to the array or group within the store. Use "" or "/" for the root group.
consolidated	logical. If TRUE, attempt to read consolidated metadata first (default).
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).

Value

A ZaroMeta object. For root group requests with consolidated metadata, the individual array metadata are attached as attr(, "consolidated").

---

Plan a read: the byte references a selection needs, without reading them

Description

Works out which chunks of an array a hyperslab touches and where each chunk's bytes are: a whole object in a native store, a byte range inside a NetCDF/HDF5/TIFF file for a virtual store (kerchunk, VirtualiZarr, byterefs), a byte range inside a shard object for a sharded array, inline bytes, or absent (reads as fill_value). No chunk data is read. For a native unsharded store no request is made at all; a sharded array reads the index of each shard it touches, and a virtual store may fetch the manifest or row groups it needs (cached in the store).

Usage

zaro_plan(
  store,
  path = "",
  start = NULL,
  count = NULL,
  meta = NULL,
  chunks = NULL,
  verbose = zaro_verbose()
)

Arguments

store	a store from zaro(), or a zaro_dataset().
path	character. Path to the array within the store.
start, count	the selection, as for zaro_read(): 0-based start and element counts in Zarr dimension order, NA count for the rest of an axis. Default the whole array.
meta	optional array metadata, as for zaro_read().
chunks	optional integer matrix of 0-based chunk grid indices (one row per chunk, Zarr dimension order), instead of start/count.
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).

Details

The result is a data frame you can inspect, filter, save or hand to another tool. Printing it summarises the cost: requests, known bytes and hosts.

Value

A zaro_plan data frame, one row per chunk, in the shape of a byterefs view (https://github.com/byterefs/spec): dim_0, ..., ⁠dim_{n-1}⁠ (the chunk's 0-based grid index, Zarr dimension order), present (FALSE for an absent chunk, read as fill_value), path (where the bytes are), offset, size (NA for a whole object of unknown size), data (list of raw, for inline chunks), then zaro's own columns: key (the Zarr key) and transport ("store", "http", "s3", "gs", "vsi", "local", "inline" or "absent": how the bytes are fetched). Attributes array (the array's path in the store) and meta record what was planned.

Examples

## Not run: 
store <- zaro(paste0("virtualizarr://https://raw.githubusercontent.com/",
  "mdsumner/virtualized/refs/heads/main/remote/ocean_temp_2023.parq"))
p <- zaro_plan(store, "temp", start = c(0, 0, 0, 0), count = c(1, 1, NA, NA))
p                          # 60 byte ranges on one host, ~4 MB
table(p$transport)

## End(Not run)

---

Read data from a Zarr array

Description

Reads a hyperslab from a Zarr array, fetching and decoding the necessary chunks and assembling them into an R array. Supports Zarr V2 and V3, including sharded V3 arrays, and every store zaro() opens. The chunks are planned with zaro_plan() and fetched together with zaro_fetch().

Usage

zaro_read(
  store,
  path = "",
  start = NULL,
  count = NULL,
  meta = NULL,
  parallel = FALSE,
  verbose = zaro_verbose(),
  assemble = TRUE
)

Arguments

store	a store object from zaro(), or a zaro_dataset() (then path is an array's name in the dataset)
path	character. Path to the array within the store.
start	integer vector. 0-based start indices for each dimension. Defaults to the origin.
count	integer vector. Number of elements to read along each dimension. Use NA for any axis to read the full extent from start. Defaults to the full extent.
meta	optional pre-fetched ZaroMeta object. If NULL (default), metadata is read from the store. Can also be the root group metadata from zaro_meta(store) - the array will be looked up by path in the consolidated entries.
parallel	logical. If TRUE, decode chunks in parallel using future.apply::future_lapply(). Fetching is always one pooled zaro_fetch() (concurrent for HTTP), so this helps only when decoding is the cost. Requires the future.apply package and a future::plan(), e.g. future::plan(future.mirai::mirai_multisession). Skipped for reads of fewer than 4 chunks. Default FALSE.
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).
assemble	logical. If TRUE (the default) return the assembled array, else the list of decoded chunks.

Value

An R array with dimensions matching count, in Zarr dimension order and named by the array's dimension names (the transpose of zaro_array() for C-order arrays). String arrays read as character.

---

Read several whole arrays in one fetch

Description

Plans every array, fetches all the plans with one zaro_fetch() (so HTTP chunks of all the arrays share one connection pool and go out concurrently, and touching byte ranges merge), then decodes each array. Each result is what zaro_read(store, path) returns. Useful for reading a dataset's coordinate arrays at open, where each array is small and the cost is the round trips.

Usage

zaro_read_arrays(store, paths, metas = NULL, verbose = zaro_verbose())

Arguments

store	a store from zaro(), or a zaro_dataset() (then paths are array names in the dataset).
paths	character vector of array paths (or dataset names).
metas	optional list of array metadata, one per path (taken from the dataset when store is one, else read).
verbose	logical. Emit diagnostic messages. Default getOption("zaro.verbose", TRUE).

Value

A named list of arrays, one per path, each with dim in Zarr order as from zaro_read(). Fetch statistics are in attr(, "stats").

Examples

## Not run: 
ds <- zaro_dataset("https://host/store.zarr")
xy <- zaro_read_arrays(ds, c("x", "y"))
attr(xy, "stats")$requests

## End(Not run)

---