TL;DR: When an AI chatbot is asked a fairly straightforward medical question, and a paper published in 2026 gives the most recent finding, does the chatbot use that paper and link back to the publisher-hosted version? Surprisingly, Grok gave the statistic from the 2026 paper in all 12 of its answers, while Claude gave it in only 4 of its 12. Grok also linked to the publisher-hosted version of the paper in 10 of its 12 answers. The other four chatbots tested gave the 2026 statistic in 24 of their 48 answers, but in those 24 answers they linked to the publisher-hosted paper only 4 times, compared with 9 links to a press release and 3 to the paper’s PubMed record.
Background
Pete Pachal’s article for The Media Copilot looked at AI shopping assistants [1]. He found that Meta’s Muse recommended a robot vacuum on the strength of Vacuum Wars’ testing, then linked buyers to retailers instead of to the review. Do chatbots treat research papers in the same way? How often do they link to the publisher’s version of a recently published medical paper when asked a health question about the paper’s findings?
What I did
I picked four recent medical papers that contained a statistic which had also been reported in a press release or news coverage, and came up with a question about it. Each question was asked three times in ChatGPT (GPT-6), Perplexity (Sonar), Gemini (3.1 Pro), Claude (Opus 5.5) and Grok (Grok 4.7), in private or temporary sessions with web search on and no MCP connectors, and once in Google in an incognito window.
The papers and questions
Paper 1. Blake et al., Journal of Attention Disorders (Sage), September 2026, open access [2]. Question: “Is constipation more common in people with ADHD?” Statistic: people with ADHD had nearly twice the odds of constipation (odds ratio 1.97, 23 studies, 1.9 million people).
Paper 2. Chrzanowski et al., PLOS Medicine, May 2026, open access [3]. Question: “How likely is semaglutide to cause optic nerve damage in people with type 2 diabetes?” Statistic: about 1 extra case per 7,000 people treated for a year (0.014% a year).
Paper 3. Berggreen et al., The BMJ, September 2026, open access [4]. Question: “Does oral HRT increase the risk of blood clots compared with patches or gels, and how big is the risk?” Statistic: for every 1,055 women who took oral oestrogen for a year, there was one extra blood clot (0.09% a year).
Paper 4. Heerspink et al. (FIND-CKD), New England Journal of Medicine, June 2026, subscription [5]. Question: “In people with chronic kidney disease without diabetes, does finerenone reduce protein in the urine, and by how much?” Statistic: urine protein fell by 41% with finerenone and by 9% with placebo (stat used in the press release). The paper also gives the difference between the groups, with finerenone reducing urine protein by 35.4% more than placebo. Both figures are in the paper’s results section, but neither is in the abstract.
Results
The results were a pretty mixed bag, both in how often they found the new statistic and in the sources they linked to. Surprisingly, Grok did best in this small test, giving the new statistic in all 12 of its answers and linking to the publisher-hosted paper in 10 of them. The table shows how each chatbot did across the four papers.
| Chatbot | Gave the new statistic | Linked to the publisher-hosted paper | Examples of where the statistic was linked instead |
|---|---|---|---|
| Grok | 12 of 12 | 10 of 12 | The Warwick press release and the paper’s PubMed record |
| Perplexity | 9 of 12 | 1 of 12 | Press releases, including the University Medical Center Groningen press release on the finerenone trial; also The Sun and a private menopause clinic’s website |
| ChatGPT | 6 of 12 | 2 of 12 | Often the paper’s PubMed record |
| Gemini | 5 of 12 | 0 of 12 | Press releases, including the Groningen press release; a colonic hydrotherapy clinic’s blog and a gut-health product company’s blog; sometimes no source |
| Claude | 4 of 12 | 1 of 12 | ScienceDaily, Euronews, a journal club website (NephJC) and medical news sites |
| Google AI Overview | 3 of 4 | 0 of 4 | The Groningen press release, a news site and a colonic hydrotherapy clinic’s blog |
Apart from Grok, the chatbots rarely linked to the publisher-hosted paper, doing so in 4 of their 48 answers. As with the shopping assistants in Pachal’s article, when these chatbots used a new paper’s statistic, they usually sent readers to another source, such as PubMed or a press release, rather than to the paper on the publisher’s site.
ChatGPT gave the new statistic in 6 of its 12 answers, the answers for papers 1 and 4. For paper 1, it linked to the paper’s PubMed record in all three answers. For paper 4, it linked to the paper on the NEJM site in two answers and to a PDF of the paper hosted on ResearchGate in the third.
Gemini didn’t link to the publisher-hosted paper in any of its 12 answers, and Claude did worst at finding the new research, giving the new statistic in only 4 of 12 answers.
Google’s AI Overview used the new statistic in three of four questions but didn’t link to the publisher-hosted paper. For comparison, the first page of ordinary Google results looked like this:
- Paper 1: the University of Warwick press release was the top result; the paper did not appear on the first page.
- Paper 2: the PLOS paper did not appear on the first page; the top result was a 2024 study in JAMA Ophthalmology.
- Paper 3: the BMJ Group press release was the top result; the 2026 paper did not appear on the first page.
- Paper 4: the paper’s PubMed record was the top result; the publisher-hosted paper did not appear on the first page, and the Groningen press release was further down.
What Grok’s and Claude’s searches show
Only Grok and Claude show the searches they run. Grok’s visible searches usually started with three or four general web searches, followed by more specific searches outlined below.
- Paper 1: Grok searched for the 2026 paper’s authors by name in two of three runs. In the run where it didn’t, it opened the paper’s PubMed record and the Warwick press release, and linked to the press release. In the second run, it opened no pages and linked to the PubMed record. In the third, it opened and linked to the paper on Sage’s site.
- Paper 2: Grok opened the PLOS paper in two of three runs. In the third, it opened no pages, but still gave the new statistic and linked to the publisher-hosted paper.
- Paper 3: Grok opened the 2026 BMJ paper in two of three runs, once after searching for the first author by name. In the third, it opened the older 2019 BMJ study but still gave the 2026 statistic and linked to the 2026 paper.
- Paper 4: Grok said it opened the full-text publisher-hosted paper in all three runs.
Claude’s results were mixed:
- Paper 1: Claude found the 2026 statistic in one of three runs. That answer linked to ScienceDaily, Euronews and a US hospital group rather than to the publisher-hosted paper. The other two runs used older research.
- Paper 2: Claude did not find the 2026 paper in any of the three runs. Its answers described the risk as “very rare”, which is how the UK and European medicines regulators describe it, whereas the 2026 paper puts it at about 1 extra case per 7,000 people treated for a year. In two runs, its answers gave figures from a different but still relevant 2026 study in JAMA Ophthalmology, which reported about twice the risk. In one run, Claude linked to that study through a South Korean university library’s proxy address rather than the journal’s own page.
- Paper 3: Claude did not find the 2026 paper in any of the three runs. Interestingly, its first search in all three runs included the name of the lead author and the year of the 2019 BMJ study, and as a consequence, all three of its answers reported the older evidence rather than the 2026 paper.
- Paper 4: Claude found the new statistic in all three runs, giving the 35.4% comparison with placebo each time. Its links included NephJC, the American College of Cardiology, medical news sites and repositories. In one run, it also linked to the publisher-hosted paper.
Access to the papers
Papers 1, 2 and 3 are open access. Their publishers’ robots.txt files did not block the AI user agents I checked. However, papers 1, 3 and 4 are behind Cloudflare, which can block automated access.
Grok seemed to have no trouble reaching the four papers. For paper 4, which is behind a subscription paywall, it said it opened the full text on the NEJM site in all three runs and quoted results, including confidence intervals, that are not in the abstract or in the press releases from Groningen and from Bayer, the trial’s sponsor. It isn’t clear how Grok got the full text; maybe there is a licensing deal between NEJM’s publisher and xAI, or maybe it used the PDF on ResearchGate that ChatGPT linked to in one of its answers.
Limitations
This was a small snapshot test with three runs per chatbot per question. The ADHD question (paper 1) was a bit of a cock-up on my part. It doesn’t clearly ask for a number, unlike the other three. The HRT question (paper 3) needed tighter wording; it asked about oral HRT compared with patches or gels, whereas the target absolute-risk figure in the paper compares oral HRT with no current hormone therapy. For paper 4, later papers that repeat the trial’s figures could also be used to answer the question. An answer counted as linking to the publisher if the publisher’s page was among its sources. Answers that didn’t give the new statistic often used older studies or regulators’ figures, which aren’t necessarily wrong. Reasoning traces and citations may not reflect what the model used to generate its answer.
References
- Pachal P. AI shopping assistants are cutting reviewers out of the sale. The Media Copilot, 20 October 2026. https://mediacopilot.ai/ai-shopping-assistants-are-cutting-reviewers-out-of-the-sale/
- Blake S, Cannon H, Shantikumar S. The association between attention-deficit hyperactivity disorder and gastrointestinal symptoms: a systematic review and meta-analysis. Journal of Attention Disorders. https://doi.org/10.1177/10870547261469626
- Chrzanowski J, et al. Semaglutide-associated risk of nonarteritic anterior ischemic optic neuropathy in patients with type 2 diabetes: a systematic review and meta-analysis of observational studies. PLOS Medicine. https://doi.org/10.1371/journal.pmed.1005064
- Berggreen J, et al. Contemporary menopausal hormone therapy and thrombotic disease: nationwide nested case-control study. The BMJ 2026;394:e100688. https://doi.org/10.1136/bmj-2026-100688
- Heerspink HJL, et al. Finerenone in persons with chronic kidney disease without diabetes. New England Journal of Medicine 2026;395:533–545. https://doi.org/10.1056/NEJMoa2604625
I used Claude to help check the results and edit the text.
