Performance of ChatGPT on USMLEPotential for AI-assisted medical education using large language models

Let's start with a gut check. If you want to know whether a general-purpose language model can really reason about medicine, you don't throw it trivia. You send it into the hardest exam doctors take in the United States—the United States Medical Licensing Examination, or USMLE. It's three steps, covering everything from biochemistry to bedside management to ethics, and the questions are built as clinical vignettes meant to test judgment, not flashcard recall. That's why Kung and colleagues picked it. If a model can survive here—without cheating, without Googling—you learn something real about its breadth, its reasoning, and whether its explanations make sense to humans. They used ChatGPT as it was then built on the GPT‑3.5 backbone: a large language model trained broadly on Internet text, with no specialty fine-tuning for medicine, and crucially, no browsing or external lookups during the test. Everything had to come from what the model already "knew." The team pulled three hundred seventy-six public questions from the June 2022 USMLE release, removed anything with images, and ended up with three hundred fifty items: one hundred nineteen from Step 1, one hundred two from Step 2 Clinical Knowledge, and one hundred twenty-two from Step 3. Each question was encoded three ways to mirror how people actually interact with these systems: open-ended, where the answer choices are stripped away; multiple choice with no forced justification; and multiple choice where the model has to explain not only its choice but why the others are wrong. A new chat session was created for every item to avoid memory carryover. Eight different people entered the prompts, then the team analyzed inputs both by individual and by medical background—physician, student, nurse, or non-medical. Before any results, they did their homework on rigor: a power calculation aiming for ninety percent power, and a two-way analysis of variance to check whether the person doing the encoding and the prompt type were causing systematic shifts. Here's the big headline. ChatGPT landed right around the USMLE's passing bar—roughly sixty percent—without any medical fine-tuning. In open-ended mode, where many humans also do better because you can reason your way to an answer without being distracted by foils, the model peaked on Step 1 at seventy-five percent. Shift to standard multiple choice on the advanced exam, Step 3, and you're in the low sixties—enough to be in the neighborhood of passing. That's not a toy result; it's a sign that a broad model can navigate detailed clinical prose and land on sensible answers much of the time. One complication the authors handled carefully is how to score "I don't know." The model sometimes gives indeterminate responses. If you censor those—treat them as skipped—the accuracy ticks up; if you include them as wrong, it drops. You can see both ends in the numbers. On Step 1 with forced justification, accuracy was sixty-four point five percent when those indeterminate answers were removed and forty-one point two percent when they were counted as incorrect. On Step 3 under the same format, the gap was smaller—sixty-five point two versus fifty-nine point eight percent—suggesting uncertainty plays out differently by step and format. The pattern tracks something familiar to human test-takers: Step 1, with its heavy basic science, tends to be the toughest when you're forced to pick from distractors; clinical steps are a bit more forgiving. But the story isn't just whether the model lands on A, B, or C. It's whether the reasoning lines up with the answer and whether that reasoning offers anything useful. Kung and colleagues formalized this with three lenses: accuracy, concordance, and insight. Concordance means the explanation actually supports the chosen answer rather than contradicting it. Across all questions and formats, ChatGPT's explanations were concordant ninety-four point six percent of the time. When the answer was right, concordance was nearly perfect—ninety-nine point one percent. When the answer was wrong, it fell to eighty-five point one percent, still high but meaningfully lower. That gap matters. It tells you that internal consistency tracks correctness, which is the kind of signal a student—or a supervising clinician using a tool—can use to calibrate trust. To make those judgments reliable, two physicians independently scored every response for accuracy, concordance, and insight, with a third doctor breaking ties when needed. Agreement between adjudicators was strong. For open-ended prompts, Cohen's kappa landed between zero point seven four and zero point eight one, which statisticians call substantial. For multiple choice formats, it was above zero point nine—near perfect. Only twenty-one items, about six percent of the set, needed arbitration. So the signals you're hearing in those concordance numbers aren't a mirage of rater noise; they held up under double review. Insight is where the educational promise shows up. The team asked a very human question: does the explanation give you a non-obvious piece of reasoning you can learn from? By their predefined criteria, at least one significant insight appeared in eighty-eight point nine percent of responses. They went further and built a density of insight score—the number of unique, valid insights divided by the number of answer choices—so a five-option question with insights for three distractors and the correct pick would score over zero point six. Across the board, that density was higher when the answer was correct: zero point four five eight versus zero point one nine nine, with a vanishingly small p-value well under one in ten thousand. In plain language, good answers came with richer teaching moments. There was one interesting wrinkle. On Step 2 Clinical Knowledge, forcing justification trimmed insight frequency by about ten percentage points compared to the no-justification multiple-choice format, and accuracy dipped in parallel. Forcing an explanation is usually helpful; here it sometimes nudged the model into overconfident, thinner reasoning on that middle step. That's a reminder that prompt design isn't neutral—it shapes how the model thinks out loud. What about stability? If different people enter the same question slightly differently, does performance wobble? The team probed that by grouping encoders by role and checking for interactions with prompt type. The result was reassuringly boring: no statistically significant encoder-by-prompt effects. Individual variation existed, but it was small next to the residual noise, and starting fresh chats each time reduced the risk of hidden carryover. In other words, the setup produced consistent behavior across users and sessions, at least within the bounds of this dataset. Now, how does a generalist model like ChatGPT stack up against a medical specialist? They compared it to PubMedGPT, a model trained on biomedical literature. On the same kind of broad clinical questions, PubMedGPT scored fifty point three percent—below the passing bar and below ChatGPT. One plausible reason, as Kung and colleagues suggest, is that journal prose is cautious and often inconclusive. A model steeped in that style may hedge too much on questions that reward synthesis and decision. By contrast, a general model that has seen everything from textbooks to patient forums to lay explanations might develop a more decisive, exam-friendly voice. There's also a data vintage issue. Older benchmarks like MedQA-USMLE, which span two thousand nine to two thousand twenty, don't fully reflect how questions are now written. The two thousand twenty-two release used here is more contemporary, and ChatGPT's broader training might simply generalize better to that style. Let's pause on what wasn't allowed, because it matters for interpreting these scores. The model couldn't browse. It couldn't plug into PubMed or UpToDate. There was no domain-specific fine-tuning, and no chain-of-thought prompting except when the format explicitly forced justification. So "near passing" here means: given only the question text, can a general model reason its way through like a diligent, reasonably prepared student? On these three hundred fifty items, the answer was often yes. There are caveats, and the authors are forthright about them. The sample is modest, which limits fine-grained analysis by topic area. Human adjudication is resource-intensive and, even with kappa in the high range, not perfect. The way you handle indeterminate answers moves the headline numbers, and we shouldn't paper over that. While they took care to ensure the questions weren't in the model's training data, we can't reconstruct exactly what the model saw during pretraining. The right response to all of this is open benchmarking: share items, share raw outputs, share scoring criteria. Kung's group did that, posting raw inputs and outputs and documenting the adjudication protocol. That makes replication, criticism, and improvement possible. So what's the practical takeaway for learners? If you're a medical student, a tool that hits around sixty percent on fresh questions and explains itself with high concordance can be a powerful study partner. You can ask it to walk through a differential or to contrast similar diagnoses, and the odds are good you'll get at least one non-obvious angle you hadn't considered. The fact that concordance drops when the answer is wrong is actually helpful: it gives you a rough confidence meter. When the reasoning tightly supports the choice, trust it more—though verify. When the logic wobbles, slow down. And beyond the classroom? Kung and colleagues, who work with AnsibleHealth, nod to early, bounded clinical uses that are basically communication problems: drafting appeal letters, simplifying radiology reports for patients, brainstorming on tough cases in a de-identified sandbox. Those are writing tasks with a medical accent. They're not diagnostic engines. Given today's numbers, that's the right scope. The model shows sparks of clinical judgment. It does not show the reliability you would demand for direct patient care decisions without guardrails. If you zoom out, there's a deeper methodological contribution here. This study didn't just check a score; it tested whether explanations align with answers and whether those explanations teach. It measured stability across users. It benchmarked against a specialist model and against contemporary questions. And it quantified the trade-offs that come with forcing a model to show its work. That's the kind of evaluation you need if we're going to move from hype to practice. Where does this go next? Carefully. Combine broad models with curated, up-to-date medical resources. Keep the evaluation open and standardized so we're not comparing apples to oranges. And always test in the context you care about—students at different levels, real clinic notes, real-time pressure—because the format changes the performance. The promise on display here is not that a chatbot can pass a licensure exam on its own. It's that, already, without specialty tuning, it can sit next to a learner and make them better, more often than not, by thinking clearly and mostly consistently out loud. That's a foothold. Build on it with rigor, and you can see where the path might lead.