The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial

What the study measured

Arms

Findings

A finding is one outcome contrast: two arms, one measure, one set of conditions. A paper is never "positive" or "negative" here; each finding carries its own direction and its own conditions. The badge that matters most is the first one: whether the AI was still available when the outcome was measured. Why that gates everything.

Limitations

Stated by the authors: course grades rather than independently administered assessments make a learning interpretation difficult researchers had no visibility into whether assignments or exams were in person, proctored online, or unproctored, nor into instructors' AI-use policies sections may differ in grading and opportunities for platform participation despite within-course random assignment LMS behavior captures only a fraction of learning activities only about 15% of treated students used the VSA; instructor encouragement may have changed GenAI use beyond the VSA, so the contrast likely reflects broader AI use induced by introducing the tool other university-supported and commercial GenAI tools were available to both arms and their use was not observed, so TOT/IV exclusion restriction may fail instructors and researchers could view student-tutor interactions, which may have shaped what students asked few instructors switched from default direct-instruction mode to tutor mode; findings concern the VSA as configured, not scaffolded AI tutoring small number of clusters; cluster-robust SEs can overstate precision (randomization inference reported as robust) student survey response 18% with differential response by arm isolated significant heterogeneity findings warrant caution given the number of interactions examined Observed by review: post-randomization removal of three treatment instructors who did not activate the VSA; the analysed sample is not the randomized sample, yet estimates are labelled ITT analysis plan registered after data collection; registered primary outcome and primary sample not reported in the paper inconsistent primary estimand: Section 5 names the full sample as primary, the abstract reports only exact-match estimates, and the Discussion contradicts Results on full-sample significance unit-of-randomization inconsistency: instructor in Section 2.2, course-section in Table 3's note; SEs clustered on sections, not instructors (Tables 1 and A6 say course level) late recruitment through week 7 with ad hoc addition of instructors to existing blocks, randomized regardless of existing assignments within the block no participant flow, opt-out counts, or outcome missingness reported final grade is instructor-posted on a 0-100 scale; with instructor-level assignment, grading leniency is confounded with treatment within the 22 exact-match clusters; exact-match control instructors were on average 15 years younger and more often graduate assistants total ELMS participation depends on activities the instructor sets in ELMS, and treated sections had fewer homework assignments at baseline, so part of the participation gap may be course design rather than student behavior treated non-users' grades resemble users' and the IV-scaled effect is implausibly large (−2.83 SD), which weakens attribution to tutor use and is consistent with section-level confounding text states humanities final-grade and letter-grade effects significant at the 1% level, but Table 5 marks both ∗∗ (p < 0.05) Discussion reports feedback-on-attempt requests as 7.6% where Section 7.3 gives 0.7% (7.6% is navigation/logistics) survey section reports two different instructor response rates (67% and 70%) in consecutive sentences randomization-inference results are asserted, not reported numerically Internal inconsistencies in the paper (pass A, merged per IN-024): the text says the humanities final-grade and letter-grade effects are significant at the 1% level, while Table 5 marks them p < 0.05; Section 5.2 says there is no significant evidence of adverse effects on final grades in the full sample, yet Table 3 marks -2.88 at p < 0.10; the Discussion gives 7.6% for requests for feedback on an attempt, whereas Section 7.3 gives 0.7%; the instructor survey response rate is given as 67% and, in the next sentence, 70%; the text says the exact-match sample has 22 instructors, while Tables A1 and A2 count 22 sections.

Who was studied

Level

university

Years

undergraduates, freshman to senior (22.6% freshmen, 28.6% sophomores, 26.6% juniors, 21.3% seniors)

Ages

mean approximately 20 years; students under 18 at study onset excluded

Country

US

Setting

University of Maryland, College Park: a public four-year university in the Mid-Atlantic United States (~30,000 undergraduates). Instructors of record volunteered; admission required a distinct ELMS course page holding most course content. Students in participating courses were enrolled unless they opted out or were under 18 (number of opt-outs not reported). Students in both conditions had free access to other university-supported GenAI tools, including ChatGPT and Gemini, and to commercial tools.

Prior knowledge

high school GPA (mean 4.373) and prior undergraduate GPA (mean 3.421) recorded and used as covariates; missing GPA set to zero with missingness indicators

Methodological notes

Extracted from: https://doi.org/10.26300/y3f8-vh05 Access: open PDF full text (EdWorkingPaper No. 26-1598, version October 2026, Annenberg Institute at Brown University), read as pdftotext -layout output data/fulltext-cache/liu-2026-fulltext.txt; figures (1-5, A1-A6) are images and not machine-readable, so figure-only content is not coded Sample-size note: full sample 2,379 students (1,227 control / 1,152 treatment), 30 instructors, 34 course-sections; exact-match sample 1,353 students (612 control / 741 treatment), 22 instructors; course letter grade observed for 2,362 (full) and 1,343 (exact-match)