What is the best AI for medical questions?

Is there a ranking of AI for health questions?

There are benchmarks, which are standard tests that many models can take. They are useful, but none of them is a final ranking. Scores change every time a company ships a new version, and each test only measures what it was built to measure.

The 2 kinds you will see most often are medical licensing exam questions, like the USMLE that US doctors must pass, and conversation tests like OpenAI's HealthBench. They measure different things, so a tool can do well on one and less well on the other.

• USMLE-style questions: multiple choice, tests medical knowledge
• HealthBench: 5,000 multi-turn health conversations scored against 48,562 rubric criteria written by 262 physicians
• HealthBench Hard: a 1,000-example subset picked because it is difficult for current models; when it launched in May 2025, the top score was 32%

What do these tests miss?

A high exam score does not mean the tool will help you in a chat. In a randomized study of 1,298 people living in the UK, published in Nature Medicine in February 2026, researchers tested GPT-4o, Llama 3 and Command R+ on 10 medical scenarios. On their own, the models named the right condition in 94.9% of cases. When people used them, those people named a relevant condition in under 34.5% of cases and picked the right next step, such as self-care or seeing a doctor, in under 44.2%. That was no better than a control group who used any source they would normally use, mostly internet search or their own knowledge.

The authors found that standard medical question tests did not predict these failures. Part of the gap was that people left out details, the AI did not always make the right answer clear, and people did not always follow its advice. They recommend testing AI with actual users before it is used for public health care.

HealthBench has limits too. Its authors say it does not measure health outcomes. Most of its conversations were synthetically generated, and a model grades the answers against the physicians' rubrics, which approximates but does not equal a doctor's judgment.

Is ChatGPT or Gemini better for medical questions?

We did not find a peer-reviewed head-to-head study of current versions that settles this, so we will not pick one. Benchmark results go stale fast because both are updated often. Note also that HealthBench was built by OpenAI, the company behind ChatGPT, although it is open source.

The bigger difference is usually between any general chatbot and a medical AI system built for health. A general chatbot is designed to answer anything. A medical system can be built to follow fixed safety rules, ask the questions a clinician would ask, and steer you to care when a warning sign shows up. HealthBench scores this behavior directly in its emergency referrals and context seeking themes.

What should I check before I trust an AI with a health question?

The World Health Organization warns that large AI models can produce false, inaccurate, biased or incomplete statements, and can lead people to trust the answer without checking it. These checks help with both problems.

• Published results: does it share test scores and methods you can read, not just claims?
• Follow-up questions: does it ask your age, how long, how bad, and what medicines you take before answering?
• Tells you when to get care: does it name warning signs and say when to see someone today or call 911?
• Privacy: does its policy say whether your messages are sold, used for ads, or used to train its models?
• Clear limits: does it say it cannot examine you, and that a clinician decides on diagnosis and prescriptions?

Where does Pymander fit?

Pymander is one option: our AI doctor is free and you reach it by text. It publishes its results at pymanderhealth.com/research. There it reports 98.2% on USMLE questions across 2 separate runs, a place in the top 5 on HealthBench Hard in a sealed, pre-registered evaluation dated September 2026, with answers and grades published, and a score of 1.0 on a 60-case safety test where missing a warning sign is dangerous.

These are Pymander's own reported results, not an independent ranking, and the limits above apply to them too. A good test score is a reason to try a tool, not a reason to skip a doctor. Pymander is not a licensed doctor and does not diagnose, treat or prescribe.

How do I get a better answer from any AI?

The Nature Medicine study suggests much of the gap comes from the conversation itself, including details people leave out. Give the AI the whole picture up front, and treat its answer as a starting point to bring to a clinician.

• Say where it is, when it started, how bad it is, and whether it is getting worse
• List your medicines, allergies and major health conditions
• Mention pregnancy, age, or anything that makes you higher risk
• Ask directly: do I need to be seen, and how soon?
• Leave out your full name, address and ID numbers unless you know how they are stored