Behavioral Evidence from Staged Multi-Criteria Disclosure: A Companion Human-Study Analysis of ML Model-Selection Deliberation in Educational QA
Outcome improvement alone does not reveal whether users actually inspected disclosed evidence or simply followed a highlighted recommendation. This companion human-study paper analyzes model-selection deliberation under a staged multi-criteria disclosure interface for educational quality assurance (QA). In the final filtered analytic sample of 38 participants and 228 completed scenarios, we examine interface telemetry, participant-level self-report, acceptance, task-level heterogeneity, and a heuristic low-engagement robustness check. Nonparametric comparisons and a clustering-adjusted GEE model were used. Disclosure uptake was selective: ranking was used in 60.5% of scenarios, weights in 47.4%, heatmap in 46.9%, and textual interpretations in 41.7%. Self-reports aligned with telemetry for four of the five major components. Improved scenarios showed longer Step 2 deliberation and greater engagement with multiple evidence surfaces, while the GEE model indicated that ranking use was significantly associated with improvement. Acceptance was favorable (34/38 preferred the agent-assisted workflow), whereas a low-engagement subgroup showed sharply reduced benefit. The observed pattern is more consistent with structured multi-surface deliberation than with shallow recommendation following.