--- title: "Beyond One Collection: Lessons from Auditing the CMR Provider Landscape" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Beyond One Collection: Lessons from Auditing the CMR Provider Landscape} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", eval = FALSE ) ``` This article is a written record of a real exploration session with `cmrlifecycle`, kept as a permanent guide rather than left to scatter across commit messages and chat logs. Every number below was produced by actually running the code shown, against the live CMR API, on 2026-08-25. The code is shown for reproducibility but is **not** re-executed when this vignette builds -- CMR's catalog changes over time, and a several-hundred-request live sweep isn't something to repeat on every `R CMD check`. For a lightweight version of these same questions re-run fresh, see the companion article `vignette("cmr-current-snapshot")`. If you arrived here only caring about one collection: `cmr_find_granules()` and `cmr_audit_revisions()` are the two functions you need, and the [README](https://github.com/hypertidy/cmrlifecycle#usage) covers them. Keep reading if you're curious why a granule-lifecycle tool grew a whole collection-discovery layer, and what that layer found. ## Summary - **This package started as a MUR-only script** and was generalized into a granule-lifecycle auditor (`cmr_find_granules()`, `cmr_audit_revisions()`) plus a collection-discovery layer (`cmr_find_collections()`, `cmr_all_collections()`, `cmr_collection_liveness()`) after the same questions kept coming up for other collections. - **A routine liveness check on `MUR-JPL-L4-GLOB-v4.1` surfaced a real, silent extraction bug.** `temporal_start` returned the *same* date for every granule in a chunk whenever the chunk's UMM-G shape was uniform -- the common case. Root cause: jsonlite flattens a single nested JSON object (not an array) into one data.frame when every row matches, instead of keeping a genuine per-granule list; the old code iterated that data.frame's columns, not its rows. This had also quietly broken this package's own MOD021KM example. Fixed, with regression tests covering both jsonlite shapes. - **Every other extraction path was individually audited** against real, uniformly-shaped CMR responses and confirmed safe -- `n_rev`, `insert_date`/`update_date`, `rev_date`, `production`, `data_url`, `s3_url`, `nc_bytes` all come from JSON *arrays* or plain scalar fields, and jsonlite never collapses those the way it collapsed `TemporalExtent`. - **Not every CMR "collection" has granules.** Some records (e.g. an AADC record, `AAS_4088_Adelie_occupancy_Mawson_Taylor`) register a single static download at the *collection* level, with zero UMM-G granules underneath. `granule_count` (from a second, lighter CMR endpoint) flags this directly. - **S3 access has a region gotcha.** `s3_url`s are only cheaply/ directly readable from compute in the *same AWS region* as the bucket (PO.DAAC is `us-west-2`), and require an Earthdata Login credential exchange regardless of region. Both are surfaced by `cmr_find_collections()`. - **`ends_at_present` reliability depends on the provider, not on CMR as a whole.** PO.DAAC leaves it `FALSE`/unset on ~95% of its ~982 collections -- including collections that are demonstrably still producing granules daily. LAADS sets it reliably on ~71% of its ~184 collections. Treat the flag as provider-dependent metadata quality, not a CMR-wide convention, and use `cmr_collection_liveness()` to check the real thing when it matters. ## The landscape: collections, granules, and this package's tools CMR (NASA's Common Metadata Repository) indexes two related but distinct kinds of record: - a **collection** is a dataset/product entry -- one row per `short_name`, e.g. `MUR-JPL-L4-GLOB-v4.1` -- carrying metadata like title, abstract, declared temporal coverage, and (for cloud-hosted data) AWS region and S3 credentials endpoint. - a **granule** is one file within a collection -- one row per `granule_ur`, e.g. one day's MUR SST grid. `cmrlifecycle`'s functions split cleanly along that line: | question | function | |---|---| | What collections exist matching a keyword/provider? | `cmr_find_collections()` | | Every collection matching a filter (or the whole catalog)? | `cmr_all_collections()` | | One page of granules for a date window (+ optional space)? | `cmr_find_granules()` | | The full revision/production lifecycle of a collection's granules? | `cmr_audit_revisions()` | | Is this collection still actually producing granules? | `cmr_collection_liveness()` | | Standard lifecycle diagnostic plots | `plot_cmr_lifecycle()` | The natural workflow is left-to-right in that table: discover a collection, decide whether you trust its stated liveness/temporal metadata, then audit its granules. Below is the narrative of how each row of that table came to exist. ## Where it started: auditing one collection The original question was narrow and practical: for `MUR-JPL-L4-GLOB-v4.1` (a daily global sea-surface-temperature grid), *when does a granule's underlying science file actually change*, as opposed to its CMR metadata being touched without the data changing? `cmr_audit_revisions()` answers this by pulling `revision-id` (a metadata-churn counter), `ProviderDates` (Insert/Update), and `DataGranule.ProductionDateTime` (the churn-immune "science file was regenerated" signal) for every granule in a date range. The first two real-world findings were about **trusting the wrong column**: ```r rev <- cmr_audit_revisions("MUR-JPL-L4-GLOB-v4.1", from = as.Date("2024-01-01")) range(rev$n_rev) #> [1] 4 23 ``` `n_rev` (CMR's `revision-id`) ranged from 4 to 23 -- but this turned out to be almost entirely **metadata churn** (bulk provider events), not evidence the underlying file changed 23 times. `production` (`DataGranule.ProductionDateTime`) is the reliable signal. A second, more subtle bug: `nc_bytes` (file size, used as a secondary staleness check) initially picked `ArchiveAndDistributionInformation[1]` as "the" file, assuming checksum sidecars (`.md5`) always came last. They don't. Re-running the full MUR audit after fixing the extraction to exclude sidecar extensions and take the largest remaining entry: ```r # before the fix: ArchiveAndDistributionInformation[1] # after the fix: exclude .md5/.cmr/.xml, take the largest remaining entry sum(nc_bytes_old != nc_bytes_fixed) / length(nc_bytes_fixed) #> [1] 0.4548 # 440 of 967 rows changed from ~95 bytes to the real ~450-780 MB size ``` **Lesson for a new user:** don't assume the first element of a UMM-G array field is the one you want. CMR's arrays aren't guaranteed ordered by role. ## Generalizing: finding out what's out there Once the granule-lifecycle tool was collection-agnostic, the next question was discovery: *what collections exist, and how do I find one without already knowing its `short_name`?* `cmr_find_collections()` wraps CMR's `collections.umm_json?keyword=...` search: ```r cmr_find_collections(keyword = "sea surface temperature", provider = "POCLOUD") #> 100 of 499 total matching collections returned -- raise `page_size` ... ``` An empty keyword turns out to match *everything*: ```r cmr_find_collections(keyword = "") #> 100 of 45431 total matching collections returned -- raise `page_size` ... ``` ...which is a fine trick, but not something to lean on by hand for a 45,000-row catalog -- `cmr_all_collections()` was built to page through that automatically (with a politeness delay between requests), landing at 45,431 collections in ~23 requests. ## Not every collection has granules Browsing that catalog surfaced `AAS_4088_Adelie_occupancy_Mawson_Taylor`, an Australian Antarctic Data Centre (AADC) record. `cmr_find_granules()` returned nothing for it: ```r cmr_find_granules("AAS_4088_Adelie_occupancy_Mawson_Taylor", as.Date("2015-01-01"), as.Date("2015-12-31")) #> NULL ``` Not a bug -- this collection has **zero UMM-G granule records at all**. It registers a single static download (`http://data.aad.gov.au/eds/4405/download`) directly on the *collection* metadata's `RelatedUrls`, a pattern common for field-survey/social-science-style datasets imported into CMR as one file rather than a time series. This motivated adding `granule_count` to `cmr_find_collections()`, via a second, lighter request to `collections.json?include_granule_counts=true` (the only CMR endpoint that reports it): ```r cmr_find_collections(short_name = "AAS_4088_Adelie_occupancy_Mawson_Taylor")[ , c("short_name", "granule_count", "temporal_start", "temporal_end")] #> # A tibble: 1 x 4 #> short_name granule_count temporal_start temporal_end #> AAS_4088_Adelie_occupancy_Mawson_Taylor 0 2015-02-15T00:… 2015-02-15T… ``` **Lesson:** `granule_count == 0` is the direct signal for a granule-less collection; don't infer it from an empty `cmr_find_granules()` call when you can ask CMR outright. For these collections, `temporal_start`/`temporal_end` from UMM-C `TemporalExtents` is the *only* date range CMR has at all. ## The S3 region gotcha `VIIRSN_L3m_CHL` (ocean chlorophyll) publishes both an `https` mirror and a direct `s3://` link: ```r x <- cmr_find_granules("VIIRSN_L3m_CHL", Sys.Date() - 100, Sys.Date()) x$s3_url[1] #> [1] "s3://ob-cumulus-prod-public/SNPP_VIIRS...9km.nc" ``` The catch: that `s3://` URL is only cheap and direct from compute running *in the same AWS region as the bucket*. `cmr_find_collections()` surfaces that region (and the credentials endpoint you need regardless of region, since `s3://` reads aren't public): ```r cmr_find_collections(short_name = "VIIRSN_L3m_CHL")[ , c("short_name", "region", "s3_credentials_endpoint")] #> # A tibble: 1 x 3 #> short_name region s3_credentials_endpoint #> VIIRSN_L3m_CHL us-west-2 https://obdaac-tea.earthdatacloud.nasa.gov/s3credentials ``` **Lesson:** check `region` *before* committing compute placement. From outside the matching region, `s3_url` is no faster than `data_url` and may incur egress charges or be blocked by bucket policy. ## A silent correctness bug hiding in plain sight Cross-checking `MUR-JPL-L4-GLOB-v4.1`'s declared collection-level `temporal_start` against a real granule sweep should have been a formality -- and it was, for the *first* granule. But every granule in the window showed the *same* date: ```r mur_early <- cmr_find_granules("MUR-JPL-L4-GLOB-v4.1", as.Date("2002-05-25"), as.Date("2002-06-10")) mur_early[order(mur_early$temporal_start), c("granule_ur", "temporal_start")] #> # before the fix -- every row identical, which is wrong: #> 1 20020601090000-... 2002-05-31T21:00:00.000Z #> 2 20020602090000-... 2002-05-31T21:00:00.000Z #> ... #> 11 20020611090000-... 2002-05-31T21:00:00.000Z ``` The cause: UMM-G's `TemporalExtent` is a single nested JSON *object* (not an array). When every granule in a chunk shares the same shape -- the common case -- `jsonlite::fromJSON(..., simplifyDataFrame = TRUE)` flattens it into **one** data.frame (one row per granule) rather than a genuine per-granule list. The old extraction code did `vapply(te, .get_temporal_start, "")`, which iterates a data.frame's *columns*, not its rows -- with only one column (`RangeDateTime`), that returned a length-1 result recycled across all 11 rows: the first granule's date, silently repeated. This wasn't a MUR-only quirk -- the exact same collapse hit the package's own MOD021KM spatial-filtering example. MOD021KM (a MODIS swath product) can produce hundreds of granules a day worldwide, so `cmr_find_granules()`/`cmr_audit_revisions()` take an optional `bounding_box`/`polygon` filter to subset to a region of interest -- for `2026-08-01`, a bounding box over eastern Australia cut 552 granules down to 8: ```r mod <- cmr_find_granules("MOD021KM", as.Date("2026-08-01"), as.Date("2026-08-01"), bounding_box = c(140, -45, 155, -35)) mod[, c("granule_ur", "temporal_start")] #> # before the fix -- all 8 granules show the identical timestamp: #> granule_ur temporal_start #> LAADS:9975746428 2026-08-01T11:15:00.000Z #> LAADS:9980651751 2026-08-01T11:15:00.000Z #> ... (6 more, all identical) ``` After the fix (`.extract_temporal_starts()`, which detects and handles both the flattened-data.frame shape and the genuine list-column shape): ```r mur_early2 <- cmr_find_granules("MUR-JPL-L4-GLOB-v4.1", as.Date("2002-05-25"), as.Date("2002-06-10")) mur_early2[order(mur_early2$temporal_start), c("granule_ur", "temporal_start")] #> # after the fix -- correctly increasing: #> 1 20020601090000-... 2002-05-31T21:00:00.000Z #> 2 20020602090000-... 2002-06-01T21:00:00.000Z #> 3 20020603090000-... 2002-06-02T21:00:00.000Z #> ... #> 11 20020611090000-... 2002-06-10T21:00:00.000Z mod2 <- cmr_find_granules("MOD021KM", as.Date("2026-08-01"), as.Date("2026-08-01"), bounding_box = c(140, -45, 155, -35)) mod2[, c("granule_ur", "temporal_start")] #> # after the fix -- 4 distinct swath-pass timestamps across the 8 granules: #> LAADS:9975746428 2026-08-01T11:15:00.000Z #> LAADS:9980651751 2026-08-01T11:15:00.000Z #> LAADS:9975763948 2026-08-01T12:50:00.000Z #> LAADS:9980653369 2026-08-01T12:50:00.000Z #> LAADS:9975774200 2026-08-01T22:15:00.000Z #> LAADS:9980668284 2026-08-01T22:15:00.000Z #> LAADS:9975768761 2026-08-01T22:20:00.000Z #> LAADS:9980660739 2026-08-01T22:20:00.000Z ``` **This didn't just mean missing dates -- it meant wrong ones**, silently. The package's own "0 NAs" check for the MOD021KM example had passed throughout, because the bug produces a value, just the *wrong* one, for every row after the first. Once found, the natural next question was: is anything else in the pipeline vulnerable to the same collapse? Every other extraction path was checked against a real, fully-uniform MUR chunk (every granule with exactly 2 `ProviderDates` entries, 6 `RelatedUrls` entries, 2 `ArchiveAndDistributionInformation` entries): ```r str(j$items$umm$ProviderDates) #> List of 6 <- genuine per-granule list, even though every #> $ :'data.frame': 2 obs. of 2 variables element has the same shape #> $ :'data.frame': 2 obs. of 2 variables #> ... ``` `ProviderDates`, `RelatedUrls`, and `DataGranule$ArchiveAndDistributionInformation` are JSON **arrays** in UMM-G, and jsonlite reliably keeps a JSON array as a genuine per-granule list-column regardless of shape uniformity -- only a single (non-array) nested *object* like `TemporalExtent` collapses. `n_rev`/`rev_date` (`meta` fields) and `production` (`DataGranule$ProductionDateTime`) are plain scalar columns on always-proper per-granule data.frames, never list-columns at all. `TemporalExtent` was the only field in the pipeline actually at risk, confirmed rather than assumed. **Lesson:** a "0 NAs" or "no error" check is not the same as a correctness check. Cross-checking a derived value against an independent source (here: the collection's own declared `temporal_start`) is what actually caught this. ## Is this collection still alive? The `ends_at_present` rabbit hole UMM-C's `TemporalExtents` carries an `EndsAtPresentFlag` -- in principle, exactly the "is this collection still growing" signal a user would want. In practice: ```r shorts <- c("MUR-JPL-L4-GLOB-v4.1", "VIIRSN_L3m_CHL", "AVHRR_OI-NCEI-L4-GLOB-v2.1", "CMC0.1deg-CMC-L4-GLOB-v3.0", "MODIS_T-JPL-L2P-v2019.0", "H09-AHI-L3C-ACSPO-v2.90") # ends_at_present, declared: #> MUR-JPL-L4-GLOB-v4.1 FALSE #> VIIRSN_L3m_CHL TRUE #> AVHRR_OI-NCEI-L4-GLOB-v2.1 FALSE #> CMC0.1deg-CMC-L4-GLOB-v3.0 FALSE #> MODIS_T-JPL-L2P-v2019.0 FALSE #> H09-AHI-L3C-ACSPO-v2.90 FALSE # granules in the last 14 days, actual: #> MUR-JPL-L4-GLOB-v4.1 14 #> AVHRR_OI-NCEI-L4-GLOB-v2.1 15 #> CMC0.1deg-CMC-L4-GLOB-v3.0 15 #> MODIS_T-JPL-L2P-v2019.0 2000 # hit the per-request page cap #> H09-AHI-L3C-ACSPO-v2.90 355 ``` Only 1 of 6 correctly declared `TRUE`. The other 5 all declared `FALSE` (with `temporal_end = NA`) despite every one having real, recent granules -- one hitting the 2000-per-request page cap outright. Six collections isn't a sample size to generalize from, though, so the same check was repeated at the scale of an entire provider's catalog, for two providers with very different pipelines: ```r poc <- cmr_all_collections(provider = "POCLOUD") # 982 collections, 2 requests table(poc$ends_at_present) #> FALSE TRUE #> 938 44 laads <- cmr_all_collections(provider = "LAADS") # 184 collections, 2 requests table(laads$ends_at_present) #> FALSE TRUE #> 54 130 ``` | provider | collections | `ends_at_present = TRUE` | ambiguous* | |---|---|---|---| | POCLOUD | 982 | 4.5% | 33.7% | | LAADS | 184 | 70.7% | 2.2% | *"ambiguous" = `FALSE`, no declared end date, but `granule_count > 0`. A random sample of 25 of PO.DAAC's 331 "ambiguous" collections found **15 (60%)** had granules landing within the last 60 days -- genuinely active, just never flagged. LAADS's own claims held up under the same scrutiny: a sample of 15 `TRUE`-declared collections was 14/15 (93%) actually active in the last 60 days; a sample of 15 "declared closed" collections was 15/15 (100%) genuinely silent past their declared end date; even LAADS's tiny 4-collection ambiguous bucket showed the same failure mode at a smaller scale (3/4 active despite the unset flag). **Lesson:** `ends_at_present` reliability is a property of *which DAAC maintains the collection's metadata pipeline*, not a CMR-wide guarantee. PO.DAAC (ocean/GHRSST products in this sample) essentially doesn't populate it; LAADS (MODIS/VIIRS atmosphere/land products) does, accurately. Don't assume either direction applies to a provider you haven't checked. This is what motivated `cmr_collection_liveness()`: rather than trust a flag whose reliability varies by provider, check real recent granule activity directly. ```r cmr_collection_liveness("MUR-JPL-L4-GLOB-v4.1") #> # A tibble: 1 x 5 #> short_name provider lookback_days recent_granule_count is_live #> MUR-JPL-L4-GLOB-v4.1 NA 60 60 TRUE # pipe-friendly, straight from a filtered collection search cmr_all_collections(provider = "POCLOUD") |> dplyr::filter(!ends_at_present, is.na(temporal_end), granule_count > 0) |> cmr_collection_liveness() ``` Even building this small helper turned up one more instance of the same general class of bug this whole session kept surfacing -- *silent* wrong output rather than a loud error. An early version passed a `NA` provider straight into the request URL, producing a literal `&provider=NA` query string that CMR quietly interpreted as "no such provider" and returned zero granules for a collection that plainly has them (60 granules in 60 days for MUR). Fixed by treating a `NA` provider as "no filter" before building the request. **Lesson, one more time:** in a codebase talking to an API this large and this heterogeneous, the failure mode to design tests against isn't "the request errors" -- it's "the request succeeds and returns a plausible-looking, wrong answer." Every bug in this narrative was exactly that shape. ## A quick-start for a new user If none of the above is what brought you here, and you just want one collection's lifecycle: ```r library(cmrlifecycle) # find a short_name if you don't already have one cmr_find_collections(keyword = "your topic here") # check it's actually alive and see its declared coverage cmr_collection_liveness("your-short-name") cmr_find_collections(short_name = "your-short-name")[ , c("temporal_start", "temporal_end", "ends_at_present", "granule_count")] # audit its granule-level revision/production lifecycle rev <- cmr_audit_revisions("your-short-name", from = as.Date("2024-01-01")) plot_cmr_lifecycle(rev) ``` See the package [README](https://github.com/hypertidy/cmrlifecycle) for the full column dictionary, spatial-filtering (`bounding_box`/ `polygon`) options for swath products, and S3-access caveats.