| Title: | A Provider-Agnostic STAC Reference Cache |
|---|---|
| Description: | Harvests SpatioTemporal Asset Catalog (STAC) discovery results into an append-only Parquet reference store: queries, acquisitions, products, and assets as separate levels, with all asset references recorded verbatim and no fixed band vocabulary. Markers and coverage are derived views, never stored state. Stores references, never pixels. |
| Authors: | Michael Sumner [aut, cre, cph] (ORCID: <https://orcid.org/0000-0002-2471-7511>) |
| Maintainer: | Michael Sumner <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.2.0 |
| Built: | 2026-10-07 08:24:12 UTC |
| Source: | https://github.com/hypertidy/starc |
Sentinel-2: platform + MGRS tile + datetime-to-seconds. Survives the
Element84 / Planetary Computer / DEA seam; reprocessings are the same
acquisition, different product rows. SAR mappers will supply
burst/frame in tile.
acquisition_id(ident)acquisition_id(ident)
Follow rel="next" links, collecting all items
fetch_all_pages(url, max_pages = 100)fetch_all_pages(url, max_pages = 100)
url |
First-page search URL (from sds::stacit()) |
max_pages |
Safety valve |
list(items = list of item lists, n_pages = int)
Harvest a region bbox into the reference store
harvest( region, provider, collection, t0, t1, store = "~/starc-store", limit = 300, pad_km = 5 )harvest( region, provider, collection, t0, t1, store = "~/starc-store", limit = 300, pad_km = 5 )
region |
One row of a regions table (region_id, lonmin, lonmax, latmin, latmax) |
provider |
sds::stacit provider token |
collection |
Collection id (must have a registered mapper) |
t0, t1
|
Query window (RFC3339 or dates sds accepts) |
store |
Root directory of the parquet store |
limit |
Page size |
pad_km |
Discovery buffer applied to the extent (see pad_extent) |
the query log row, invisibly
The deterministic sibling of harvest(): same store, same tables,
same query-log-last discipline. region_id in the query log is
"mgrs:", keeping tile-month harvests distinguishable from
region-bbox harvests while sharing every derived view.
harvest_mgrs( mgrs, month, collection = "sentinel-2-c1-l2a", endpoint = "https://earth-search.aws.element84.com/v1/search", store = "~/starc-store", limit = 300 )harvest_mgrs( mgrs, month, collection = "sentinel-2-c1-l2a", endpoint = "https://earth-search.aws.element84.com/v1/search", store = "~/starc-store", limit = 300 )
mgrs |
MGRS code, e.g. "43DDE" (no "MGRS-" prefix) |
month |
"YYYY-MM" |
collection |
Collection id |
endpoint |
STAC search endpoint (raw URL; this mode constructs the query itself rather than going through sds::stacit) |
store |
Root directory of the parquet store |
limit |
Page size |
the query log row, invisibly
A mapper declares product_family and two functions: identity() returns the fields that name the physical acquisition across providers; promote() returns properties promoted to product columns. Everything else in the item is preserved via the raw stash and the properties JSON column.
mappersmappers
Marker: max solarday per collection (deduped, derived, never stored)
marker(store = "~/starc-store")marker(store = "~/starc-store")
Joins products -> queries via query_id to recover region_id. Product rows from harvests predating the query_id column drop out of the inner join (re-harvest to backfill).
marker_region(store = "~/starc-store")marker_region(store = "~/starc-store")
Build a grid:code + month search URL
mgrs_month_url( mgrs, month, collection = "sentinel-2-c1-l2a", endpoint = "https://earth-search.aws.element84.com/v1/search", limit = 300 )mgrs_month_url( mgrs, month, collection = "sentinel-2-c1-l2a", endpoint = "https://earth-search.aws.element84.com/v1/search", limit = 300 )
mgrs |
MGRS code, e.g. "43DDE" (no "MGRS-" prefix) |
month |
"YYYY-MM" |
collection |
Collection id |
endpoint |
STAC search endpoint (raw URL; this mode constructs the query itself rather than going through sds::stacit) |
limit |
Page size |
Region extents are member-centroid bboxes, so singleton and co-located regions degenerate to points; discovery queries need area. Pads by min_km on every side, longitude scaled by cos(lat) so the pad is honest at 77S. This is a discovery buffer, not the site window: scene footprints are ~100 km, so modest padding suffices to catch every scene touching the sites.
pad_extent(extent, min_km = 5)pad_extent(extent, min_km = 5)
Solar day at the item centroid
solarday_at(datetime, centroid_lon)solarday_at(datetime, centroid_lon)
Same-zone pairs intersect by interval arithmetic (both rectangles in one CRS). Cross-zone pairs reproject the buffered scene extent into the tile's zone via densified project_extent() – the corner-only trap applies to MGRS squares exactly as it did to Heard's bbox.
tiles_to_mgrs(tiles, valid, lat_bands)tiles_to_mgrs(tiles, valid, lat_bands)
tiles |
data.frame from tiles_for_extent2() (tile_id, zone_id, res, col, row) |
valid |
Character vector of valid S2 MGRS codes (see starc's inst/extdata/valid_mgrs_sentinel2.parquet) |
lat_bands |
Latitude band letters to consider (derive from the tiles' latitude range; e.g. c("C", "D", "E") around 53S-77S) |
long data.frame: tile_id, mgrs, same_zone, offset60 (scene-origin easting mod 60 in the tile's CRS frame – whether 60 m assets land lattice-aligned or sub-pixel shifted)