Package: blocklist 0.0.1.9002

blocklist: Harvest Byte-References of Compressed Array Data Chunks Native to HDF5-Accessible Files

Generate byte-reference stores from chunked array storage, on disk, on s3 storage or at urls. Array sources are logically mosaiced with gdal multidimensional tool 'mdim mosaic' and this is used to extract and organize raw byte references (path, offset, size) written to Parquet with semantic grouping of manually-sharded, positionally-indexed parquet reference store. The structure is a directory of parquet files, manually partitioned by variable (subdirectory) and by flat-chunk-index range ('refs.N.parq', 'record_size' rows each), with the partition keys implied positionally rather than stored as columns (i.e. _not_ hive-encoded). This is compatible with "kerchunk-parquet Zarr", and is used as an generic and intermediate convenience before generating 'icechunk' versions.

Authors:Michael Sumner [aut, cre]