Skip to contents

Export an Aeme object's model output to a partitioned Parquet store

Source: R/write_aeme_parquet.R
write_aeme_parquet.Rd

Writes the long-format model output already loaded onto aeme (see load_output) into a Hive-partitioned Parquet store suitable for querying across many lakes at once (e.g. with DuckDB), plus updates a single lakes.parquet dimension table describing every lake exported into out_root so far.

Usage

write_aeme_parquet(
  aeme,
  out_root,
  model,
  quantiles = c(0.05, 0.25, 0.5, 0.75, 0.95),
  compression = "gzip",
  depths = NULL,
  depth_ref = c("surface", "bottom")
)

Arguments

aeme

Aeme object with output loaded (see load_output).

out_root

character; root directory of the store (created if missing).

model

character vector of model codes to export (see list_models). Default: every model with output on aeme.

quantiles

numeric vector of quantiles (in [0, 1]) for the ensemble summary tier. Default c(0.05, 0.25, 0.5, 0.75, 0.95).

compression

character; Parquet compression codec, passed to arrow::write_parquet(). Default "gzip" – slower to write than arrow's own default ("snappy") but meaningfully smaller on disk for this kind of repetitive numeric time series, which matters more here than write speed since a store is written occasionally and read often. Use "snappy" instead if write/decode speed matters more than size for your case, or "zstd" for a middle ground.

depths

numeric vector of depths (m) at which to summarise depth-resolved variables in output_summary, referenced to depth_ref. NULL (default) summarises at each member's own depths – see Details. Only the summary tier is affected; output_raw always holds the stored depths, in metres below the surface.

depth_ref

character; what depths is measured from. "surface" (default): metres below the water surface. "bottom": metres above the lake bed, so 1 is the water 1 m above the bottom, found from each member's water level (LKE_lvlwtr) on each date. Requires depths. Stored in the depth_ref column of output_summary.

Value

(Invisibly) a list with lake_row (the dimension-table row just written), raw_files, and summary_files (paths written; empty if the summary tier was skipped because aeme only has one ensemble member).

Details

Two output tiers are written per model, partitioned as <tier>/lake_id=<id>/model=<model>/part-0.parquet:

  • output_raw: every ensemble member's full time series (date, ens, var_sim, depth, value) – for drill-down into a single lake (e.g. a spaghetti plot across ensemble members).

  • output_summary: value collapsed across the ensemble dimension at each date/var_sim/depth into mean, sd, n_ens, and the requested quantiles (named q05, q50, ... from quantiles) – for overview views that scan many lakes at once, so the ensemble collapse doesn't need recomputing on every load.

Depth-resolved variables

AEME stores depth-resolved output on a grid of fractions of the water depth scaled to the lake level on each day, so the absolute depths differ between ensemble members whose water level differs (and drift as a member's level moves). output_raw keeps that grid as it is. Collapsing across members at those depths would leave most groups with one or two members, so by default output_summary is only really useful for variables with no depth dimension. Supply depths to compute the summary of depth-resolved variables at fixed depths instead: each member's profile is linearly interpolated to the requested depths on every date before the members are collapsed. A depth that falls outside a member's water column on a date (deeper than the lake, or above the surface) is left out for that member, so n_ens says how many members contributed. output_summary is only written when aeme has more than one ensemble member: with a single member, collapsing across ens is a no-op that would just re-store the same values under 8 columns (mean, sd, n_ens, 5 quantile columns) instead of 1 (value) – pure overhead, no new information. Readers should check a lake's n_ens in lakes.parquet (see read_aeme_parquet) and fall back to output_raw for any lake where it's 1.

Re-running this for one lake only rewrites that lake's own partition directories and its one row in lakes.parquet; other lakes already exported into out_root are untouched. If a lake's ensemble count changes between exports (e.g. it grows from 1 to many members), any stale output_summary partition for that lake/model is removed or (re)written to match.

Examples

if (FALSE) { # \dontrun{
aeme_file <- system.file("extdata/aeme.rds", package = "AEME")
aeme <- readRDS(aeme_file)
write_aeme_parquet(aeme, out_root = file.path(tempdir(), "parquet"))

# Summarise depth-resolved variables 1 m and 5 m below the surface, and 1 m
# above the lake bed
write_aeme_parquet(aeme, out_root = file.path(tempdir(), "parquet_depths"),
                   depths = c(1, 5))
write_aeme_parquet(aeme, out_root = file.path(tempdir(), "parquet_bottom"),
                   depths = 1, depth_ref = "bottom")
} # }