AI Learning Evidence

Not guidance. A record of published research on AI-assisted learning, and our assessments of it. The limits

Unrestricted use of a general-purpose LLM chatbot during practice improves university students' subsequent performance on assessments taken without AI.

University students · Unassisted performance after assisted practice

Current status
Insufficient evidenceEvidence unclear · what the nine statuses mean
Why
Evidence leans against the claim — a significant post-use quality deficit vs web search, and an underpowered retention null — but nothing adequately powered tests it directly, and the largest randomized offer cannot (partial AI at its exam). Pass B read the same evidence as NOT_SUPPORTED; the conservative status stands until something powered exists.
What would change it
Any adequately sized university RCT with unassisted post-tests and an active comparison.
Linked evidence
8 links · 2 contradicts · 5 consistent, but doesn't test the claim · 1 no direct evidence
Last updated
Sep 18, 2026

The status is the collection's own judgment on its nine-label scale — not a certainty rating, not a GRADE level, and not advice. Every status change is dated, reasoned, and kept below under History.

01The evidence

The evidence, sorted by what it shows

Contradicts (2)

Consistent, but doesn't test the claim (5)

  • ChatGPT's impact on student learning outcomes: a meta-analysis of 35 experimental studiesMeta-analysis · 2026 · Study tier 3 · Secondary partial · University partial

    Assisted/unassisted pooling plus Critically-low appraisal confidence; higher-education subgroup shares the same problems. Rests on finding F1, judged at assessment v2.

  • How Generative AI Influences Learning Outcomes in Programming Education: A Three-level Bayesian Meta-analysisMeta-analysis · 2026 · Not rated · Secondary unclear · University unclear

    Same pooled IPO result against the university-scoped unrestricted-chat claim: direction positive, measured-without-AI by category definition, but unstratified by level or design in the accessible text and pooled across heterogeneous configurations — cannot test; retained for completeness per the wu-2026 meta precedent. Rests on finding F2, judged at assessment v1.

  • The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University TrialCluster-randomized trial · 2026 · Preliminary · Secondary different · University direct · Working paper

    The paper's own stated primary analysis (full sample, 34 sections): final course grade -2.88 points on 0-100 with controls (SE 1.60, p < 0.10; control SD 10.10), with the full-sample letter grade (F3) flat at +0.02 GPA. Population is a direct university match and the arm is coded vanilla_chat with unrestricted access, so the spec 11.3 lint does not refuse a link. It still cannot test the claim, on three separate counts. (1) Gate: the grade is an instructor-posted composite of semester work whose AI-access conditions the authors could not see (availability not_reported, timing during_intervention), so it is not an "assessment taken without AI" and per Rule 2 cannot support or contradict a learning claim. (2) Contrast: what was randomized is the instructor switching on a course-grounded, scope-restricted RAG assistant in the LMS sidebar; only ~15% of offered students used it, and control students kept free ChatGPT/Gemini access, so the estimate is access-to-this-tool against background GenAI availability, not unrestricted chatbot use against none. (3) Measure: instructor-level assignment confounds grading standards with treatment. Pinned so the claim page carries the primary estimate alongside the F2 subsample signal rather than the subsample alone (the abstract's selective emphasis, assessment limitation 3). Same treatment as nie-2025 F4: a randomized offer that cannot test the claim, recorded with the reason. Rests on finding F1, judged at assessment v1.

  • The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University TrialCluster-randomized trial · 2026 · Preliminary · Secondary different · University direct · Working paper

    A course-grade harm signal, exact-match sample only: among sections of the same course (22 clusters, 1,353 students), final grade -4.12 points with controls (SE 1.80, p < 0.05; 0.37 control SD), letter grade -0.11 GPA (F4), homework score not detectably different (F6). It fails the gate exactly as F1 does (AI conditions on graded work not_reported, during_intervention), so it cannot contradict the claim, however its direction sits with the body's existing lean. Weight is further limited by the study itself: PRELIMINARY (working paper); exact-match control instructors averaged 15 years younger and were more often graduate assistants, so grading leniency is confounded with arm; three treatment instructors were removed after randomization; the analysis plan was registered after data collection; no participant flow; and the implied IV effect (-2.83 control SD) is implausible as an effect of tool use, with treated non-users graded like users - pointing to section-level differences rather than tutor use. Best read, per the assessment, as: offering this tool as configured did not raise course performance. The intervention is also indirect for "general-purpose" chat (RAG over instructor materials, course-scope restriction, instructor-visible prompts), though as delivered it was mostly direct-answer mode. Rests on finding F2, judged at assessment v1.

  • The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement But May Increase Adopters' Exam PerformancesRandomized trial · 2025 · Study tier 2 · Secondary different · University partial · Partly vendor funded

    The largest randomized offer cannot test the claim: the unproctored exam had partial AI availability and High-RoB intervention-affected missingness; population match partial (adult global MOOC). Rests on finding F4, judged at assessment v2.

No direct evidence (1)

02Certainty

Certainty by outcome

A claim can be broken into separate bodies of evidence — one per outcome, split by whether AI was available at assessment and when the outcome was measured. Each body carries four separate judgments: how confident we are (certainty), what the evidence points to (the conclusion), how directly it speaks to this claim (applicability), and who has stood behind the judgment. Confidence and conclusion are never merged into one word.

UA-ANY — Unassisted performance, immediate and delayed · AI at assessment: no · immediate post

Field Assembly certainty: Weak (not a GRADE rating — what our scale means) · Conclusion: Inconsistent · Applicability: Partial · AI: two passes agreed

Rated against: any improvement vs comparator

No benefit demonstrated; one significant post-use quality deficit and one underpowered retention null trending pro-ChatGPT

DomainJudgment and reasoning
Risk of biasserious · a LOW trial (N=33, unsettled D4) and a Some-concerns lab experiment
Inconsistencyserious · a significant quality deficit vs an underpowered null trending the other way — the two results disagree in kind
Indirectnessserious · single-session/single-quiz designs; neither is sustained practice
Imprecisionvery serious · N=33 and N=91; the null is underpowered by the authors' own analysis
Reporting and publication biasnot serious · not assessable at two studies

Population: university students (medical; general German university)
Comparator: non-AI resources (institutional materials; Google search)
Assessed by software, two independent passes · search: seed/corpus.yaml (verified inventory, proposal section 7) + data/searches/ · method: EVIDENCE-MODEL.md v2 + fa-certainty-scale v2

2 studies in this body. 3 further records were considered and left out:

03History

How this assessment has changed

  • Insufficient evidenceSep 18, 2026 · Evidence unclear

    initial assessed status (Q-009, owner-accepted IN-018)

  • Not yet assessedSep 18, 2026 · Evidence unclear

    initial curated status