Secondary students with low prior knowledge gain less, or are harmed more, by AI assistance during practice than high-prior-knowledge peers.
- Current status
- Preliminary signalEvidence unclear · what the nine statuses mean
- Why
- Two independent weak analyses point the claim's way (retention gains confined to high-prior learners; a positive treatment-by-baseline interaction), while the strongest trial's preregistered heterogeneity analysis — gated for stance because its assessment conditions are not reported — found no detectable ability moderation. An early, internally inconsistent signal.
- What would change it
- Prespecified prior-knowledge moderation analyses on unassisted outcomes in the larger randomized studies.
- Linked evidence
- 3 links · 2 supports · 1 consistent, but doesn't test the claim
- Last updated
- Sep 18, 2026
The status is the collection's own judgment on its nine-label scale — not a certainty rating, not a GRADE level, and not advice. Every status change is dated, reasoned, and kept below under History.
The evidence, sorted by what it shows
Supports (2)
- From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in NigeriaRandomized trial · 2025 · Preliminary · Secondary direct · University different · Working paper
Treatment-by-baseline interaction +0.151 SD (p<0.05) on the unassisted endline: higher-baseline students gained more. Weak support: PRELIMINARY working paper; attrition assessed-not-settled. Rests on finding F8, judged at assessment v2.
- Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory ProgrammingRandomized trial · 2023 · Study tier 2 · Secondary direct · University different
High-prior learners retained significantly more at one week with Codex training; low-prior did not. Unprespecified, figure-derived subgroup split; RoB assessed-not-settled (D2). Rests on finding F10, judged at assessment v2.
Consistent, but doesn't test the claim (1)
- Generative AI without guardrails can harm learning: Evidence from high school mathematicsCluster-randomized trial · 2025 · Study tier 1 · Secondary direct · University different
Preregistered heterogeneity analyses found limited-to-no ability moderation — but AI availability at assessment is not_reported for this finding, so per the gate it cannot support or contradict; recorded as context. Rests on finding F5, judged at assessment v2.
Certainty by outcome
A claim can be broken into separate bodies of evidence — one per outcome, split by whether AI was available at assessment and when the outcome was measured. Each body carries four separate judgments: how confident we are (certainty), what the evidence points to (the conclusion), how directly it speaks to this claim (applicability), and who has stood behind the judgment. Confidence and conclusion are never merged into one word.
UA-MOD — Prior-knowledge moderation, unassisted outcomes · AI at assessment: no · immediate post
Field Assembly certainty: Weak (not a GRADE rating — what our scale means) · Conclusion: Inconsistent · Applicability: Partial · AI: two passes agreed
Rated against: differential favoring high-prior learners
Two weak signals that gains concentrate among higher-prior learners; the strongest trial detected no ability moderation
| Domain | Judgment and reasoning |
|---|---|
| Risk of bias | serious · unprespecified figure-derived subgroup split; working paper with unsettled attrition |
| Inconsistency | serious · two weak supporting analyses against a preregistered no-detectable-heterogeneity result (gated for stance, informative here) |
| Indirectness | serious · different subjects, tools, and moderator definitions |
| Imprecision | very serious · subgroup analyses of small samples |
| Reporting and publication bias | not serious · not assessable |
Population: secondary students (programming camps; Nigerian secondary schools)
Comparator: same practice/schooling without AI
Assessed by software, two independent passes · search: seed/corpus.yaml (verified inventory, proposal section 7) + data/searches/ · method: EVIDENCE-MODEL.md v2 + fa-certainty-scale v2
2 studies in this body. 1 further record was considered and left out:
- Generative AI without guardrails can harm learning: Evidence from high school mathematics
heterogeneity finding's AI availability not_reported (gate); preregistered no-moderation result retained as context
How this assessment has changed
- Preliminary signalSep 18, 2026 · Evidence unclear
initial assessed status (Q-009, owner-accepted IN-018)
- Not yet assessedSep 18, 2026 · Evidence unclear
initial curated status