eval: select measured ERP configurations for Top-K dashboards - #602

Why

The dashboard comparison used hardcoded ERP parameters, so its memory and planning-time results did not test ERP selection. The original results also had event-order, query-endpoint and exact-retention accounting errors.

What

• Replace the fixed configuration with persisted measured error/resource records and the real ASAPPlanner ERP selector.
• Match empirical cardinality, event rate and rank-mass descriptors; minimize retained memory under common per-instance and deployment budgets.
• Compare shared versus independent 30-second pane layouts, and independently search each AutoSketch window with discrete LHS and numeric neighbors.
• Align all methods to the same 400 held-out query endpoints, replay individual events in order, and record per-query timings and accuracy.
• Withdraw v1 results and provide scripts to regenerate profiles, synthetic/Google trials, validation, report and figures.

How

The offline backend window benchmark adapter emits sketch-bench ERP-v1-compatible records with per-window Recall loss. Synthetic profile seeds are independent of evaluation seeds. Custom profiling only reads calibration data; its cold-start construction cost is reported separately. Planner filters measured accuracy and selects minimum retained-memory configurations. Missing evidence goes to an explicitly labeled exact fallback because a frequency additive-error bound does not certify Top-K recall.

Before this PR

The ERP arm always selected CMS ×ばつ512 with heap 32; the reported microsecond planning time timed a hardcoded filter. The raw v1 results are withdrawn and remain available only in git history.

After this PR

Changing a profile's measured error changes the selected configuration; oversized deployments are rejected. ERP can choose shared or independent configurations, with selected record IDs, match distance and composed costs recorded. All baselines answer windows ending at panes 121–220.

Completed release-mode measurements (median retained logical memory):

Synthetic ERP retains about 26% less memory. Google selects identical configurations and does not establish SLA compliance; its calibration evidence is insufficient to guarantee held-out accuracy. No thresholds were tuned on held-out results to hide this failure.

The evaluation report includes workload/query/data definitions, planning time for every baseline, separate profile-construction costs, update/merge/readout measurements, and figures. Adjacent committed files contain raw results, measured profiles, replay data, and provenance. Google trials repeat the same trace for timing, not independent statistical evidence.

Verification

• Five Rust regression tests passed: measured selection, memory caps, shape mismatch, discrete LHS, boundary ties and common dashboard endpoints.
• Completed measured Zipf, uniform and Google profile construction, plus three release-mode evaluation trials each for synthetic and Google data.
• Validated 12,000 per-query samples, common endpoints, timing sums, reported violation counts and approximate-method memory constraints; regenerated and visually checked figures.
• ERP planning includes catalog loading; offline construction is reported separately. Timed operation sums are not end-to-end client latency.

Scope

The adapter produces measured window-conditioned ERP evidence; it does not invoke the sketch-bench executable. The benchmark calls the real Planner ERP selector, not the full production compiler. Arbitrary pane-width search, production drift handling and RSS accounting remain outside this implementation. Memory uses a common retained-payload proxy; timings are wall-clock measurements. The Google evaluation remains the explicitly identified sparse 1,971-event interval from one shard. Accuracy failures will be reported as failing points, not accuracy-equivalent wins.