AI Tutors Work Only If Students Use Them: Trial
A two-year trial across 18 middle schools found Khan Academy's AI tutor raised math scores modestly - but here's the catch: the gains looked just like those from practice without any AI at all. The real story isn't whether AI tutoring works; it's that 96% of students tried it and almost none used it when they actually needed help.
Automation needs a narrow first win
The best first AI workflow is usually a repeated task with a clear input, clear output, and a human approval step.
Does an AI tutor actually help kids learn math? A large, careful trial just gave us one of the clearest answers yet - and the answer is more interesting than a simple yes or no.
Researchers ran a two-year cluster randomized trial across 18 Tennessee middle schools, testing Khan Academy's AI tutor, Khanmigo, during daily remedial math sessions. The tutor was deliberately configured to coach rather than hand over answers. Students who were assigned to it gained about 1.3 national percentile ranks per term in math achievement - roughly 0.06 to 0.08 standard deviations over a school year.
What The Numbers Actually Show
On paper, that's a real, measurable gain. But here's the detail that changes how you should read it: the gains from using Khanmigo looked a lot like the gains students get from practicing on Khan Academy without any AI assistance at all.
In other words, the trial didn't show that the AI tutor added something special on top of regular practice. It showed results consistent with practice itself. A full year of active participation implied an effect of 0.14 standard deviations - respectable for a school intervention, but nothing like the transformative leap the AI tutoring hype has promised.
Worth a look
Phugialy Picks
OMOTON C2 Adjustable Aluminum Phone Stand
We'd buy this if: You want your phone upright on your desk for calls/notifications without a cable getting in the way.
We'd skip this if: You never keep your phone on your desk.
Screen Mom Screen Cleaner Spray (16oz)
We'd buy this if: You've been wiping your monitor with your shirt sleeve.
We'd skip this if: You already have a cleaning routine sorted.
Anker USB-C to USB-C 60W Fast Charging Cable (2-Pack)
We'd buy this if: You need a reliable spare charging cable that won't fray in a month.
We'd skip this if: You already have enough cables lying around.
Some Phugialy Picks use affiliate links. If you buy through one, Phugialy may earn a commission. It doesn't change what we recommend. Full disclosure →
The Part Most Coverage Will Skip
It's easy to read this as "AI tutoring doesn't work." That's not quite what happened, and the difference matters.
Ninety-six percent of students tried Khanmigo at least once. Access wasn't the problem - the tool was there, and kids used it. But the median student messaged it on only about a third of the days they practiced. And in exercise sessions where they actually made a mistake, students engaged with the tutor substantively in just 17 percent of them.
That last number is the one to sit with. An AI tutor can only coach a student who talks to it. If a student gets a problem wrong and doesn't reach out for help, there's nothing for the tutor to do - no matter how good its pedagogy is.
Engagement Is The Constraint, Not Access
The researchers behind this trial put it plainly: the binding constraint appears to be engagement - realizing the promise of AI tutoring will require getting students to use it, not just giving them access.
Here's my read: most of the conversation about AI tutors treats deployment as the finish line. Build the tool, configure it to teach rather than answer, roll it out - done. This trial suggests that's actually the starting line. The hard problem isn't making an AI that can explain fractions; we've largely done that. The hard problem is designing something a struggling middle schooler genuinely wants to turn to after getting a problem wrong.
That reframes what "working on it" means for anyone building educational AI. Progress won't come only from better models - it will come from whatever gets that 17 percent number up.
The one-sentence version: we've proven AI tutors can be available and harmless; we haven't yet proven kids will use them when it counts.
Got a question about how this applies to you? →
If you're curious what an AI tutor would need to look like to earn that engagement, see "Beyond the Chatbot: Building a Socratic AI That Actually Teaches."
Keep reading
Follow the thread
Spectral Attacks Reveal a Hidden Weak Spot in Vision-Language Models
Researchers found a way to attack vision-language models using the model's own internal geometry — aligning representations with the subspaces the model cares about least. It beats existing baselines, and the same insight could make these models tougher.
Read this noteSame lane, different angle
SPEAR Turns Any Unreal Project Into an Embodied AI Simulator
A new open-source library exposes over 14,000 Unreal Engine functions to Python at no wrapper cost — turning any UE project into candidate training infrastructure for embodied AI at 73 fps photorealistic rendering.
MADB: Where Music AI Actually Stands on Human Judgment
Models asked to judge music like humans do fall substantially short — and for once we know because someone built the benchmark first instead of shipping demos.