Design space: expose ISTP uncertainty (DELTA_VAR) and quality flags on SpeasyVariable #335

@jeandet

Context

Speasy currently reads ISTP CDF/NetCDF variables and their DEPEND_* axes, but drops two
categories of metadata that ISTP-family datasets carry and that users routinely need for serious
scientific use: per-sample uncertainty and per-sample quality flags. Today both end up
either silently discarded or, at best, sitting unused as a name inside meta.

This issue documents how each is actually stored in ISTP-compliant files (verified against the
ISTP Metadata Guidelines, not from
memory), what Speasy would need to fetch each one, and 2-3 design options per mechanism. The goal
is to agree on direction before writing a spec/plan for implementation — the two are standardized
to very different degrees, so they may not want the same shape of solution.

Shared finding: pyistp can't reach these variables today

pyistp.loader.ISTPLoader.data_variable(name) only returns variables where the master CDF's
VAR_TYPE (or CSA's PARAMETER_TYPE) is data (ISTPLoaderImpl._update_data_vars_lis). Both
mechanisms below point at, or live in, support_data variables, which are invisible to that API.
Any of these designs needs one of:

• a pyistp addition, e.g. ISTPLoader.support_data_variable(name) mirroring data_variable
  but without the VAR_TYPE filter (same master/skeleton fallback, same sig_digits/Cluster
  quirks already handled in _get_axis), or
• Speasy reaching past pyistp into pycdfpp/the netCDF driver directly for these specific
  variables, which duplicates logic pyistp already has (axis resolution, master-CDF fallback,
  ISTP-noncompliance warnings) and would drift from it over time.

The first option is very likely correct regardless of which design below is picked, since both
need "give me this named support_data variable's values + attributes."

Shared finding: SpeasyVariable has no free-form extension point

SpeasyVariable and DataContainer (speasy/products/variable.py,
speasy/core/data_containers.py) are __slots__-based, and every operation that transforms a
variable (view, copy, select, filter_columns, to_dictionary/from_dictionary, __eq__,
the numpy __array_function__/__array_ufunc__ overloads) is written against the current fixed
set of slots. Whatever new data we attach (uncertainty array, quality array) must be threaded
through all of those explicitly — it can't be smuggled into meta (a plain dict, not
resliced/copied by any of the above) or bolted on dynamically.

---

1. Uncertainty / confidence interval — DELTA_PLUS_VAR / DELTA_MINUS_VAR

Standardization: formal, structural. ISTP defines both as optional variable attributes:
"point to a variable (or variables) which stores the uncertainty in (or range of) the original
variable's value," applied as a (+/-) around the value. Both attributes must reference a variable
in the same CDF; when the uncertainty is symmetric, both often point to the same companion
variable. This is the only one of the two that's a real, generic, machine-followable pointer —
no per-mission convention to reverse-engineer.

Design options:

• A. Auto-attach whenever present. No opt-in kwarg: if DELTA_PLUS_VAR/DELTA_MINUS_VAR are
  in a variable's attributes, always resolve and attach them. Simplest for users, but changes the
  shape/cost of every existing get_data() call for datasets that happen to carry these attributes
  (extra file variable to fetch, extra memory) — a silent behavior change for existing callers.
• B. Opt-in kwarg, e.g. get_data(..., with_uncertainty=True) (recommended). Matches the
  "opt-in extra kwargs" shape already in mind: spz.get_data() already threads **kwargs through
  request_dispatch.py down to the codec's load_variable(s), so a new kwarg is cheap to add at
  that layer and only costs anything for callers who ask for it. When absent, and the delta
  attributes exist, still surface that they exist somewhere cheap (see storage section) so users
  discover the feature without paying for it.
• C. Separate accessor method, e.g. variable.load_uncertainty(). Lazily fetches on demand
  from the still-open remote file reference. Avoids fetching data the user never asked for even
  under option A's discovery model, but requires keeping a handle to the source file/URL alive
  after the initial load, which the current codec interface (load_variable returns a
  self-contained SpeasyVariable, no source handle) doesn't do — bigger structural change.

Storage/exposure options (independent of the fetch decision above):

• Add a slot to SpeasyVariable, e.g. uncertainty_plus / uncertainty_minus, each itself a
  DataContainer sharing the parent's axes — analogous to how DataContainer already carries
  meta. This is the most natural fit for "new mechanism in the SpeasyVariable to store/expose
  those," and reuses the existing shape/broadcast rules instead of inventing new ones.
  view/select/copy etc. would each need to also transform these companions the same way they
  transform values.
  • Single symmetric case (delta_plus is delta_minus) is common; worth special-casing to avoid
    holding a duplicate array.
• Alternative: a generic variable.companions: Dict[str, DataContainer] slot instead of two named
  ones, so quality flags (§2) can reuse the exact same plumbing rather than adding another pair of
  named slots. Trades a little discoverability (.uncertainty_plus vs .companions["DELTA_PLUS_VAR"])
  for one mechanism instead of N.

---

2. Quality flags — QUALITY_FLAG / QUALITY_BITMASK (convention, not ISTP-formal)

Standardization: informal, mission-specific. There is no ISTP variable attribute that points
from a data variable to its quality companion (nothing like DELTA_PLUS_VAR exists for quality).
What exists in practice is a widely — but not universally — used naming convention, most visibly
across Solar Orbiter instruments (RPW, MAG, SWA): a QUALITY_FLAG variable (CDF_UINT1,
human-readable 0-4: bad / known-problems / survey-only / good-for-publication / excellent) and a
QUALITY_BITMASK variable (CDF_UINT2, per-bit machine flags whose meaning is mission-specific
and documented only in free-text CATDESC/VAR_NOTES, since ISTP has nothing equivalent to CF's
flag_meanings/flag_values). Other missions use other names (quality, *_qual, flag, THEMIS
and Cluster both have their own schemes). There's no flag_meanings-equivalent attribute to parse
bit semantics generically — that part stays free text no matter what Speasy does.

Consequence: unlike uncertainty, this can't be "always resolve the pointer" — there is no
pointer. Discovery has to be either name-based heuristics (fragile, mission-specific) or explicit
configuration (e.g. an inventory-side mapping of data variable → quality variable name, similar
to how direct_archive/generic_archive already carry per-dataset YAML config).

Design options:

• A. Heuristic name matching, opt-in kwarg with_quality=True (or similar). On request, probe
  the source file's variable list for {var}_QUALITY_FLAG, {var}_QUALITY_BITMASK,
  QUALITY_FLAG, QUALITY_BITMASK, quality, etc., in a fixed priority order, and attach whatever
  matches first. Works for the common case for free; silently attaches nothing (not an error) for
  the missions that don't follow the convention, or attaches the wrong thing for a mission whose
  naming coincidentally collides — needs a warning-on-attach so users know what got matched, and a
  narrow enough pattern list is essential.
• B. Config-driven per-provider/per-dataset mapping. Extend the existing inventory/config
  mechanisms (CDAWeb master CDF inspection, or a YAML side-table like generic_archive's) to record
  the actual quality-variable name once it's known for a dataset, instead of guessing at
  request-time. More reliable, more upfront work, and needs a place to seed/maintain that table
  (community contributions? scraped from master CDFs at inventory-build time by checking for
  VAR_NOTES/CATDESC mentioning "quality"?).
  • The DEPEND_0 that the quality variable itself carries is a real, cheap validity check: if it
    matches the data variable's time axis it's very likely a true per-sample companion; if it
    doesn't, reject the match rather than attaching a misaligned array. Worth applying to option A
    too.
• C. Don't build discovery into Speasy at all; document the pattern and let user code fetch the
  companion variable explicitly (e.g. spz.get_data("cda/quality_flag/SOLO_L2_RPW-...") if it's
  independently indexed) — punts the whole problem, but is honest about how unstandardized this is,
  and costs nothing to build or maintain.

Recommendation leaning: start with A + the DEPEND_0 validity check, reusing whatever
storage slot §1 lands on (companions["quality"] or a dedicated slot), since it's the only option
that ships something useful without a large new subsystem. B is a natural follow-up once real usage
shows which datasets need explicit overrides.

---

Cross-cutting open questions

1. Does the opt-in kwarg belong at the spz.get_data() level (provider-agnostic, threaded through
   request_dispatch.py), at the ISTP codec level only, or both (codec-level flag,
   provider-level passthrough)? Non-ISTP providers (AMDA, SSC) have no equivalent concept today.
2. Companion-variable storage: one generic companions: Dict[str, DataContainer] slot (uncertainty
   • quality reuse it) vs. named slots (uncertainty_plus/uncertainty_minus/quality) — affects
     discoverability vs. how much new plumbing every future companion type needs.
3. Should attaching a quality/uncertainty companion be cached/keyed separately from the base
   variable request (so asking with vs without the kwarg doesn't create two incompatible cache
   entries for "the same" product)?
4. pyistp change (support_data_variable(name)) needs to happen in that separate repo/release
   before any of this lands in Speasy — worth filing there first regardless of which options are
   chosen here.