Learning gains from AI-assisted practice persist on delayed (one week or longer) retention tests taken without AI.
- Current status
- Preliminary signalEvidence improving · what the nine statuses mean
- Why
- The claim now has its first substantial gate-passing delayed evidence: a MODERATE multicentre trial with precise tool-free advantages at 7 days and ~16 weeks for one guardrailed, verification-heavy configuration. Against it stands kalam-2025's severely underpowered one-week null from an unguarded single-episode design — heterogeneity with a candidate explanation in recorded moderators (instructional design and exposure), not established contradiction. PRELIMINARY_SIGNAL rather than anything stronger because the supporting evidence is one non-randomized trial, the effect attaches to a bundled configuration, the instrument is author-developed without construct validation, and the subject sits outside the concentration areas.
- What would change it
- Delayed unassisted retention arms in university studies at meaningful sample sizes.
- Linked evidence
- 3 links · 2 supports · 1 contradicts
- Last updated
- Oct 1, 2026
The status is the collection's own judgment on its nine-label scale — not a certainty rating, not a GRADE level, and not advice. Every status change is dated, reasoned, and kept below under History.
The evidence, sorted by what it shows
Supports (2)
- Independent Engineering Transfer After Traceable Generative-AI-Assisted Learning: A Six-University Controlled Trial with Deterministic Cluster Allocation in Agricultural Engineering EducationQuasi-experimental study · 2026 · Study tier 2 · Secondary different · University direct
Direct, gate-passing test of the claim at ~16 weeks: supervised tool-free transfer advantage of +2.34/32 [1.54, 3.14], d=0.51, robust to adverse missing-data shifts through 12 points, with no credible decay from the 7-day measurement. University population direct. Tempered in weight, not stance: allocation was deterministic (model-based CIs), the effect attaches to a complete guardrailed traceable configuration rather than AI access per se, and the subject (agricultural engineering) sits outside the concentration subjects — an indirectness note for the body, not a bar to the stance. Rests on finding F2, judged at assessment v1.
- Independent Engineering Transfer After Traceable Generative-AI-Assisted Learning: A Six-University Controlled Trial with Deterministic Cluster Allocation in Agricultural Engineering EducationQuasi-experimental study · 2026 · Study tier 2 · Secondary different · University direct
A second, earlier delayed timepoint: +2.78/32 [2.02, 3.55], d=0.72 at exactly 7 days — the claim's 'one week or longer' boundary, which IN-002 codes delayed_post. Same strengths and tempering as F2; linking both timepoints lets the claim page show persistence across the interval rather than a single snapshot. Rests on finding F1, judged at assessment v1.
Contradicts (1)
- ChatGPT as a Learning Tool for Medical Students: Results From a Randomized Controlled TrialRandomized trial · 2025 · Study tier 3 · Secondary different · University partial
The only delayed unassisted university measure: no demonstrated persistence at one week; severely underpowered. Rests on finding F5, judged at assessment v2.
Certainty by outcome
A claim can be broken into separate bodies of evidence — one per outcome, split by whether AI was available at assessment and when the outcome was measured. Each body carries four separate judgments: how confident we are (certainty), what the evidence points to (the conclusion), how directly it speaks to this claim (applicability), and who has stood behind the judgment. Confidence and conclusion are never merged into one word.
UA-DEL — Unassisted retention at >=1 week · AI at assessment: no · delayed post
Field Assembly certainty: Weak (not a GRADE rating — what our scale means) · Conclusion: Varies by intervention design · Applicability: Partial · AI: two passes agreed
Rated against: persistent advantage
Durable tool-free advantage (d 0.72 at 7 days, 0.51 at ~16 weeks) under one guardrailed verification-bound configuration; underpowered null under an unguarded single episode
| Domain | Judgment and reasoning |
|---|---|
| Risk of bias | serious · One MODERATE quasi trial (deterministic post-baseline allocation; otherwise strong conduct: active control, masked duplicate scoring, cluster-appropriate analysis) and one LOW trial with an unsettled RoB domain. |
| Inconsistency | not serious · Directions differ (sustained benefit vs null), but the two results differ on recorded moderators — guardrailed sustained configuration with mandatory verification vs unguarded single lookup episode — and the null is severely underpowered; heterogeneity with a candidate design explanation, not established contradiction, hence varies_by_design rather than inconsistent. |
| Indirectness | serious · Partial throughout: agricultural engineering and a medical quiz rather than concentration subjects; one specific bounded configuration (indirect for unguarded-chat questions); single-country, single-language settings. |
| Imprecision | serious · syromyatnikov-2026 is precise (CI half-width ~0.77 on 32 points at both timepoints); kalam-2025 contributes almost no information (~43% power). Downgraded from very_serious now that one precise result exists; still serious with a single informative study. |
| Reporting and publication bias | not serious · Not assessable with two heterogeneous studies; neither was prospectively registered in a public registry; no field-wide registry exists. |
Population: university students (medical quiz episode; agricultural engineering module)
Comparator: non-AI resources / matched active control
Assessed by software, two independent passes · search: seed/corpus.yaml (verified inventory, proposal section 7) + data/searches/ (monthly-sweep-2026-09-30.md) · method: EVIDENCE-MODEL.md v2 + fa-certainty-scale v2
2 studies in this body.
How this assessment has changed
- Preliminary signalOct 1, 2026 · Evidence improving
IN-022 owner-accepted status change (Q-017)
- Insufficient evidenceSep 18, 2026 · Evidence stable
initial assessed status (Q-009, owner-accepted IN-018)
- Not yet assessedSep 18, 2026 · Evidence unclear
initial curated status