It depends on the test: among the published results in the table below, the top HealthBench Hard score is 46.2% for OpenAI's gpt-5-thinking, and the top MedQA score is 91.1% for Google's Med-Gemini. Pymander reports 33.3% on HealthBench Hard and 98.2% on text-only official USMLE sample questions, both under its own test settings. Most of these numbers come from the company that built the AI, and they were measured with different setups and dates, so they are not one fair ranking. A high exam score shows medical knowledge, not how well a tool handles your actual conversation.
What do the main medical AI benchmarks measure?
Most published results come from 2 kinds of tests. Exam-style tests ask multiple-choice questions with 1 right answer. Conversation tests grade a full written reply to a person's health message.
HealthBench, published by OpenAI in May 2025, has 5,000 multi-turn health conversations. Each reply is graded against rubrics written by 262 physicians, with 48,562 rubric items in total, and a model (GPT-4.1) does the grading. HealthBench Hard is a 1,000-conversation subset picked because frontier models scored poorly on it.
MedQA is a research dataset of USMLE-style multiple-choice questions. The GPT-4 paper from Microsoft and OpenAI calls it an unofficial, third-party source. The official USMLE sample questions are a separate, smaller set published by the exam's own organizers. A score on one is not the same as a score on the other.
- HealthBench and HealthBench Hard: open-ended replies to health conversations, scored 0 to 100% against physician rubrics.
- MedQA (USMLE-style): percent of multiple-choice questions answered correctly.
- Official USMLE sample questions: percent correct against the official answer key.
What are the published HealthBench Hard scores?
These are the HealthBench Hard results we could check on the primary source. Higher is better. When HealthBench launched in May 2025, OpenAI wrote that no model it tested scored above 32%. Newer models released in 2026 are not listed, because we could not confirm their scores on the developer's own pages.
- OpenAI gpt-5-thinking: 46.2%. Reported by OpenAI, GPT-5 System Card, August 2025.
- OpenAI gpt-5-thinking-mini: 40.3%. Reported by OpenAI, GPT-5 System Card, August 2025.
- Pymander (a pipeline built around Claude Opus 4.8): 33.3% (mean 0.3331, N=999 of 1,000, 95% CI plus or minus 1.96 points), under 4 test-only settings described below. Reported by Pymander, which describes it as a sealed, pre-registered run, September 11, 2026, Zenodo record 22714759.
- OpenAI o3: 31.6%. Reported by OpenAI, GPT-5 System Card, August 2025. The May 2025 HealthBench paper gives 0.32.
- OpenAI gpt-5-main: 25.5%. Reported by OpenAI, GPT-5 System Card, August 2025.
- OpenAI GPT-4o: 0.0%. Reported by OpenAI, GPT-5 System Card, August 2025.
What are the published HealthBench scores?
On the full 5,000-conversation HealthBench, the original OpenAI paper reported scores from 16% for GPT-3.5 Turbo to 60% for o3 in May 2025. The same paper also tested Claude 3.7 Sonnet, Gemini 2.5 Pro, Grok 3 and Llama 4 Maverick, but showed their scores in charts, so we have left those numbers out.
- OpenAI o3: 60%. Reported by OpenAI, HealthBench paper (arXiv 2505.08775), May 2025.
- OpenAI GPT-4o: 32%. Reported by OpenAI, HealthBench paper, May 2025.
- OpenAI GPT-3.5 Turbo: 16%. Reported by OpenAI, HealthBench paper, May 2025.
What are the published MedQA and USMLE scores?
The GPT-4 paper says GPT-4 exceeded the USMLE passing score by over 20 points. Note that the question sets differ: MedQA results and official USMLE sample results should not be compared row by row.
- Google Med-Gemini: 91.1% on MedQA (USMLE). Reported by Google Research and Google DeepMind, arXiv 2404.18416, April 2024.
- Google Med-PaLM 2: up to 86.5% on MedQA. Reported by Google authors, Nature Medicine, January 8, 2025.
- OpenAI GPT-4: 81.4% on MedQA (US, 4-option, 5-shot). Reported by Microsoft and OpenAI authors, arXiv 2303.13375, March 2023.
- Google Flan-PaLM: 67.6% on MedQA. Reported by Google authors, Nature, July 12, 2023.
- Pymander: 98.2% on official USMLE sample questions (479 of 488 answers, 244 text-only questions run twice). Reported by Pymander, August 2026.
- OpenAI GPT-4: 86.7% on the official USMLE Sample Exam and 86.65% on the USMLE Self-Assessment (5-shot, per the paper's tables). Reported by Microsoft and OpenAI authors, arXiv 2303.13375, March 2023.
Why can't you rank these scores as one list?
Each number was measured with its own prompts, settings and date, and almost all were reported by the company that built the system. None of the rows above is an independent, head-to-head test run by a third party.
HealthBench was built by OpenAI. Its Hard subset was chosen as the conversations with the lowest average score across o3, Grok 3, Gemini 2.5 Pro, Claude 3.7 Sonnet and Llama 4 Maverick, so those models were tested on cases picked because they did poorly. A model, not a person, grades every reply.
Pymander's HealthBench Hard run was a test setup with 4 disclosed changes from the product people text, including allowing pediatric and specific dosing answers, longer replies, and an extra prompt asking for complete answers. A setup closer to the shipping product scored 12.2%. Pymander also used 140 of the 1,000 conversations to tune its setup beforehand. On the 859 it never used, it scored 32.8%. The paper says its raw answers and grades will be in a public repository, but the address is listed as "announced at release", so outside checks are not yet possible.
Pymander's USMLE run left out questions that need an image. GPT-4's USMLE Sample Exam score includes 49 questions that refer to images or other media, out of 376. On the text-only questions, the GPT-4 paper reports 87.8% (5-shot). The 2 question sets may not be identical.
When should you get care?
In an emergency, call 911No benchmark score means an AI can replace an in-person check. If a symptom is severe, getting worse, or worries you, see a clinician the same day, even if an AI said it could wait.
Call 911 for anything that feels like an emergency. If you are thinking about hurting yourself, call or text 988.
How our AI doctor decides when to send you to care ›Does a high exam score mean an AI is good at health advice?
Not on its own. In an exam question, the facts you need are all in the question and 1 listed answer is correct. In a chat with a person, the AI has to ask the right questions, notice what was left out, and say when to get care. Pymander's own USMLE write-up says a closed exam score does not, by itself, extend to open-ended care.
Google's AMIE study, published in Nature in April 2025, tested a conversation AI in 159 text-chat case scenarios with validated patient-actors, compared with 20 primary care doctors. Specialist physicians rated AMIE better on 30 of 32 measures. The authors say the results should be read with caution and that more research is needed before use in clinics.
In a pilot study with questions from routine care, Med-PaLM 2's authors report that specialist physicians' answers were still preferred overall over the AI's answers. Benchmarks are a useful starting point to compare tools, not proof that a tool is safe for you.
Common questions
Which AI scores best on medical benchmarks?
It depends on the benchmark. Of the published results we could check on primary sources, OpenAI's gpt-5-thinking has the top HealthBench Hard score at 46.2%, and Google's Med-Gemini has the top MedQA score at 91.1%. These were reported by the companies that built them, at different times and with different setups, so no single winner is proven.
What did Pymander score on medical benchmarks?
Pymander reports 98.2% on text-only official USMLE sample questions across 2 runs (479 of 488 answers) and a mean of 0.3331 (33.3%) on HealthBench Hard, N=999, in a run it describes as sealed and pre-registered, dated September 11, 2026. That HealthBench Hard run used 4 test-only settings, and a setup closer to the shipping product scored 0.1220. These are Pymander's own results, not an independent ranking.
What is HealthBench Hard?
It is a set of 1,000 conversations from OpenAI's HealthBench, chosen because frontier models scored low on them. Replies are graded against physician-written rubrics by a GPT-4.1 grader. When it launched in May 2025, no tested model scored above 32%.
Is MedQA the same as the USMLE?
No. MedQA is a research dataset of USMLE-style questions, described in the GPT-4 paper as an unofficial, third-party source. The official USMLE sample questions are a smaller set published by the exam's organizers. Scores on the 2 are not directly comparable.
Are AI benchmark scores independent?
Mostly not. Nearly every number on this page was reported by the company that built the AI, and HealthBench itself was made by OpenAI, although it is open source. Look for published methods and outputs that others can check.
Pymander is not a replacement for a physician and does not provide medical advice, diagnosis, or treatment. Always seek the advice of a qualified health provider with questions about a medical condition. If you think you may have a medical emergency, call 911.