Release 1.1: Batch Rework, Custom Individuals & Intervention Overhaul #85

Welcome to the 1.1 release! This release rebuilds the batch API around a streaming, memory-bounded processing model, adds support for custom per-individual attributes, and overhauls the intervention engine for type-stable, allocation-free execution of measures.

Note that the batch rewrite and parts of the intervention internals contain breaking changes. Please read the relevant sections below before upgrading.

Batch Rework

The Batch workflow has been redesigned so that memory no longer scales with the number of runs. Instead of instantiating and retaining a Simulation (and later a ResultData) per run, a Batch now stores lightweight simulation configurations and folds each run's results into running accumulators as it goes.

• Streaming, memory-bounded processing: BatchProcessor now aggregates results using Welford online statistics (mean, variance, min/max, 95% CI). Each run is executed, accumulated, and discarded, so peak memory is proportional to a single run rather than to the size of the batch. A new WelfordState accumulator and helpers (welford_update!, welford_to_aggregate, welford_df_to_stats_df, welford_df_to_stats_df_multicol) back this model and produce the same aggregate schema as the existing aggregate_* utilities.
• Configuration-based Batch: Batch now holds a vector of simulation-config NamedTuples and per-run setup functions instead of Simulation objects. Create one with Batch(n_runs = N; simargs...), where simargs are any keyword arguments accepted by Simulation(). An optional setup = sim -> ... hook lets you attach interventions, strategies, or triggers to each run after construction but before it executes.
• process! replaces run!'s old semantics: process!(batch; ...) runs every configuration sequentially and returns a BatchProcessor. run!(::Batch) is retained as an alias for process! (it now returns a BatchProcessor rather than mutating and returning the Batch).
• Per-simulation seeding: Every run receives a deterministic seed derived from a single master seed. Pass seed = N to process!/BatchData for fully reproducible batches, and retrieve the seed actually used via seed(bp).
• Representative (median) run: Pass median_by = pp -> ... to select a criterion (e.g. total infections); the run whose criterion is closest to the median across the batch is re-run in full and stored as median_run(bp). For multi-group batches, one median run is computed per group. Disable with median_by = nothing (the default).
• Per-label / per-group aggregation: Pass group_by = :label (or any config field) to accumulate statistics separately per group, accessible via per_label(bd) / per_group. gemsplot(bd) then draws one mean±CI ribbon per label automatically.
• BatchData-native plots: gemsplot(bd::BatchData) now dispatches directly to mean±CI ribbon plots built from the streamed accumulators — no need to retain every per-run ResultData. BatchData(b) remains a convenient shorthand for BatchData(process!(b)).
• keep_rundata for individual runs: Individual run traces are retained by default in the lightweight LightRD format and are available via runs(bd). Pass keep_rundata = false to drop them entirely for minimum memory.
• customlogger passthrough: process!(batch; customlogger = ...) attaches an independent copy of a CustomLogger to each run, so logged data is never shared across runs.
• New accessors: BatchProcessor/BatchData gain cumulative_cases, dark_figure, generation_times, n_runs, r0, median_run, seed, and more.

Breaking changes

• Batch no longer stores Simulation objects. Use Batch(n_runs = N, kwarg = val, ...) and simconfigs(batch) / simsetups(batch).
• simulations(batch) has been removed; use simconfigs(batch).
• add!(::Simulation, ::Batch) and remove!(::Simulation, ::Batch) have been removed; use add!(::NamedTuple, ::Batch).
• ResultData(::Batch) and PostProcessor(::Batch) have been removed; use BatchData(::Batch).
• gemsplot's splitlabel is renamed splitgroup, and the combined = :bylabel option is renamed combined = :bygroup. The old names are no longer exported.
• customlogger!(::Batch, ::CustomLogger) has been removed; pass customlogger to process! instead.
• Several BatchProcessor accessors that required storing every ResultData (config_files, pathogens, settingdata, population_pyramid, setting_age_contacts, strategies, run_ids, runtime, allocations, extract, extract_unique, ...) have been removed. Use simconfigs(batch) for configuration metadata, or process with keep_rundata = true and read per-run data via rundata(bp).

See the [batch tutorial](https://immidd.github.io/GEMS/stable/tut_batches/) for the updated workflow and migration examples.

Custom Individual Extensions

You can now attach arbitrary per-agent attributes to Individuals without modifying the core struct, by passing ind_extension to the Population or Simulation constructor. Extension fields behave exactly like built-in fields, and every existing GEMS function that accepts an ::Individual continues to work unchanged.

• Three ways to define extensions: (1) named columns — pass a vector of column names that already exist in the population DataFrame (e.g. ind_extension = [:my_attr]), whose values are read from each row and wrapped automatically; (2) a separate extension DataFrame joined by id (individuals absent from the table receive zero-filled values with a warning); (3) a factory function ind -> ext that computes each individual's extension from its base attributes — it may return any struct, including a @kwdef struct constructed by keyword arguments when the struct has many fields.
• Transparent field access: Custom attributes are read and written like core fields (ind.my_attr, ind.my_attr = 0.9). Access is implemented via getproperty/setproperty! overloads; for literal core-field names the dispatch is constant-folded and inlines to a single getfield, so built-in field access keeps its original performance. Extension data is held in a boxed extensions::Any slot, wrapped in an AutoExtension NamedTuple for the named-column/DataFrame paths.
• Population from a DataFrame: Simulation's population argument now also accepts a DataFrame directly (in addition to a path, identifier, or Population object), and Population(df; ind_extension = ...) builds individuals in parallel as before.
• Round-trips to DataFrame: dataframe(population) appends extension fields as additional columns after the base columns, taking the schema from the first extended individual.
• Collision safety: Extension field names must not clash with core Individual fields. Collisions are rejected at load time with an explicit error, preventing a custom field from silently shadowing a built-in attribute and keeping the population exportable.
• Type-stability caveat: Because the extensions slot is typed Any, reading custom fields is type-unstable. This is negligible in dynamically-dispatched code such as custom transmission_probability or sample_contacts! methods, but for heavy per-contact computation on custom fields you can recover full type stability with a function barrier.

See the [Custom Individual Extensions](https://immidd.github.io/GEMS/stable/tut_configfiles/#Custom-Individual-Extension) section of the config-file tutorial.

Intervention Engine Optimization

The intervention pipeline — the event queue, strategy callbacks, and per-tick setting iteration — has been reworked to be type-stable and allocation-free on the hot path. This adds a dependency on FunctionWrappers.jl.

• Tick-bucketed event queue: EventQueue has been rewritten from a sorted-vector priority queue (with O(N) searchsortedfirst + insert!) into a "calendar" queue of per-tick buckets. enqueue! is now an O(1) push! into the relevant tick bucket, and the queue is drained in tick order via a forward-advancing head pointer. New peektick and empty! (capacity-retaining) functions are provided; first is no longer part of the public queue API (use peek).
• Type-stable strategy callbacks: Strategy conditions, delays, and the per-measure process_measure calls are now wrapped in concrete FunctionWrapper types (IPredicate/SPredicate, IDelayFn/SDelayFn, IProcessFn/SProcessFn). MeasureEntry is now parameterized on the focal object type (Individual or Setting), keeping each strategy's measure vector concretely typed. The process_measure callback for each measure is built once at add_measure! time — where the concrete measure type is statically known — and stored on the event, turning the former dynamic process_measure dispatch into a fixed indirect call. IMeasureEvent/SMeasureEvent are now immutable structs carrying this prebuilt callback.
• Type-stable setting iteration: SettingsContainer now stores each setting type in a concretely-typed Vector{T} (the dict value type was widened from Vector{Setting} to Vector). A new foreach_setting_vector iterates the built-in setting types (BUILTIN_SETTING_TYPES) with full type stability and no dynamic dispatch, and a new settings(sim, ::Type{T}) accessor returns a typed Vector{T}. step!, open!, close!, and the tick triggers now route through these type-stable paths via function barriers.
• Compile-time debug switch: All @debug logging on the intervention hot path is now gated behind the INTERVENTION_DEBUG constant (default false), so the logging machinery is fully elided in normal runs. Flip it to true when debugging.
• FindMembers fast path: The selectionfilter is now wrapped as an IPredicate with a has_filter flag and a _select_all sentinel default. When no custom filter is supplied, the filtering pass is skipped entirely, and process_measure no longer rebuilds the member list multiple times.

Note for API users

condition(strategy) and condition(measure_entry) now return a wrapped predicate (IPredicate/SPredicate) rather than a bare Function. The wrappers are callable exactly like the original closures, but code that asserted condition(str) isa Function or compared the stored condition by identity will need updating.

Bugfixes

• Disease progression InexactError: Transition-time calculations across all progression categories (Asymptomatic, Symptomatic, Severe, Hospitalized, Critical) now use round(Int16, ...) instead of Int16(...). Previously, a sampled transition duration that wasn't already a whole number would throw an InexactError; the values are now rounded to the nearest tick. death_probability is also explicitly cast to Float64 before comparison, and DiseaseProgression's default keyword values are now Int16-typed.

Other Changes & API Cleanup

• Internal PostProcessor constructor removed: The undocumented 7-argument PostProcessor(simulation, population, infections, vaccinations, deaths, tests, quarantines) reconstruction constructor has been removed.
• Expanded test coverage: This release adds substantial new tests across interventions, settings, populations, simulations, batches, reporting, and post-processing — including the new tick-bucketed event queue, type-stable accessors, the Dict-based Simulation constructor, and the batch streaming pipeline.