litlas
Log inCreate account
AI workflow

Using AI to Summarize a Paper PDF: Check for Overgeneralisation

Tools that search the literature before answering have largely closed off invented references. They have not closed off misreading a real paper, and the measured form of that misreading is overgeneralisation. Here is what to hand the model, the three places to check its output, and where the responsibility sits.

By the Litlas editorial teamReviewed by the Litlas editorial team

This article was drafted with AI assistance and checked by a human editor before publication.

Most advice about AI summaries of papers is about invented references. That was the right worry in 2023, and it is largely the wrong one now: the services built for this search the literature first and answer from what they retrieved, which removes the step where a citation could be invented.

The failure that survives is less conspicuous. The model retrieves a real paper, cites it correctly, and then generalises slightly beyond what the paper said. That is the thing to check, and it has a measured shape.

Three ways an AI summary breaks, and the one retrieval cannot prevent

Consensus is a service built around retrieval-first search over the literature. It splits the problem into three on its own site [2]:

  1. Fake sources — the model cites a paper that does not exist.
  2. Wrong facts — the model answers from memory, with no source, and is simply incorrect.
  3. Misread sources — the model summarises a real paper, cites it, and gets it wrong.

Consensus argues the first two are structurally excluded by its architecture, because search happens before generation and every cited paper is a retrieved one. Then it concedes the third, in its own words: “no AI system is perfect. Sometimes a model can misinterpret a paper and summarize it incorrectly and this can happen in Consensus.”

That concession is the honest state of the art. Retrieval fixes where the citation came from. It does not fix what the model said the citation showed.

Ungrounded chatbots are a different matter, and the old worry still applies to them. Walters and Wilder had ChatGPT produce literature reviews on 42 topics and checked all 636 resulting citations [8]: 55% of the GPT-3.5 citations and 18% of the GPT-4 citations were fabricated outright. If you are pasting a question into a general chatbot with no retrieval, check that every reference exists before you check anything else.

The measured shape of a misreading is overgeneralisation

Peters and Chin-Yee tested ten models, including ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.3 70B and Claude 3.7 Sonnet, comparing 4,900 generated summaries against the scientific texts they came from [5]. Three results are worth carrying around:

  • Models generalise beyond the original even when told not to. Explicitly prompting for accuracy did not remove the effect. DeepSeek, ChatGPT-4o and LLaMA 3.3 70B overgeneralised in 26–73% of cases.
  • They do it far more than people do. Against human-authored summaries of the same work, LLM summaries were nearly five times more likely to contain broad generalisations (odds ratio 4.85, 95% CI 3.06–7.70).
  • Newer is not safer. Newer models tended to score worse on generalisation accuracy than earlier ones.

Concretely, overgeneralisation looks like a hedge going missing. “Was associated with” becomes “reduces”. “In this cohort of 240 adults” becomes “in adults”. “May suggest” becomes “shows”. Each edit is small, each is defensible in isolation, and the stack of them turns a careful finding into a claim the authors did not make.

Hand it the body, not the abstract

The abstract is the tempting thing to paste, because it is short and free. It is also the part of the paper least safe to summarise from.

Pitkin and colleagues compared abstracts against their own articles across six major medical journals [6] and found data that were inconsistent with the body, or absent from it altogether, in 18–68% of abstracts depending on the journal. Boutron and colleagues went further and looked at trials whose primary outcome was not statistically significant [1]: 58.3% of the abstract conclusions contained spin, meaning reporting choices that present the treatment favourably anyway.

So an abstract-only summary compounds two distortions: the authors’ framing comes first, and the model’s generalisation is added on top of it.

Give it the full text, then narrow. Tang and colleagues found that models “struggle to identify the salient information and are more error-prone when summarizing over longer textual contexts” [7], so the whole PDF is not automatically the best input either. In practice:

  • Ask about one section at a time when the answer has to be exact — methods, or the results table, or the limitations.
  • Ask it to quote before it concludes: “quote the sentence that supports each claim you make.”
  • Ask for the sample and the setting explicitly, since those are what disappear first.

Three things to check in the output

  1. The scope words. Compare the summary’s subject with the paper’s population. If the paper studied 240 adults in one hospital and the summary says “patients”, the scope moved.
  2. The verbs. Association verbs must survive: associated with, correlated with, predicted, and the like. A causal verb in the summary needs a causal design in the paper.
  3. The numbers, against the table. Tang and colleagues also found that automatic quality metrics do not strongly correlate with how good a summary actually is [7], which is a general warning: nothing scores this for you. Open the table.

All three checks send you back to the source text rather than the summary. When that source is in a language you do not read fluently and the checking is what costs you the time, reading a research paper in a second language covers getting through a first one end to end.

What you are still responsible for

If any of this reaches a manuscript, the responsibility does not move. The ICMJE recommendations put it in one sentence [3]: “humans are responsible for any submitted material that included the use of AI-assisted technologies.” Nature Portfolio’s policy says the same thing structurally [4] — “Human accountability is non-transferable” — and asks authors to “remain fully accountable for originality, accuracy, and integrity.”

Read those as a description of what you are signing, not as a warning label.

Getting the paper in front of your AI without a setup step

The friction in all of this is mechanical: find the paper, get the actual text out of it, open the model, paste, and write the prompt again.

Litlas has a ✨ button that collapses that into one click. It takes the text of the paper you are looking at — a saved web page or a document you are drafting works too — and opens ChatGPT, Claude, Gemini or Gemini Notebook in a new tab with the prompt already prepared. Right-click or long-press the button to change what it sends. Because the handoff is per paper, you can put the same paper in front of two different models and compare what they say about the same methods section, which is a cheap way to notice a generalisation that only one of them made.

Papers arrive there from one search across 16 scholarly sources (OpenAlex, Crossref, DataCite, OpenCitations, ROR/ORCID, arXiv, DBLP, OpenReview, ACL Anthology, PubMed, PMC Open Access, DOAJ, CORE, OpenAIRE Graph, Unpaywall and Common Crawl), and what you keep goes into a board with your own notes attached, exporting as BibTeX, RIS or CSL-JSON.

The free plan needs no card and works right away.

→ Try Litlas now

Three things to remember

  1. Retrieval-first services have largely closed off invented citations. They have not closed off misreading a real paper, and the vendors say so themselves.
  2. The measured failure is overgeneralisation, and prompting for accuracy does not remove it. Check the scope words and the verbs against the paper’s own sentences.
  3. Summarising the abstract stacks the model’s generalisation on top of the authors’ framing. Work from the body.

Bibliography

  1. Boutron, Isabelle, Susan Dutton, Philippe Ravaud, and Douglas G. Altman. “Reporting and Interpretation of Randomized Controlled Trials with Statistically Nonsignificant Results for Primary Outcomes.” JAMA 303, no. 20 (2010): 2058–2064. https://doi.org/10.1001/jama.2010.651
  2. Consensus. “How Consensus Works.” https://framer.consensus.app/home/blog/how-consensus-works/. Accessed 22 August 2026.
  3. ICMJE. “Use of Artificial Intelligence by Authors.” Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. https://www.icmje.org/recommendations/browse/artificial-intelligence/ai-use-by-authors.html. Accessed 22 August 2026.
  4. Nature Portfolio. “Artificial Intelligence (AI).” Editorial Policies. https://www.nature.com/nature-portfolio/editorial-policies/ai. Accessed 22 August 2026.
  5. Peters, Uwe, and Benjamin Chin-Yee. “Generalization Bias in Large Language Model Summarization of Scientific Research.” Royal Society Open Science 12, no. 4 (2025): 241776. https://doi.org/10.1098/rsos.241776
  6. Pitkin, Roy M., Mary Ann Branagan, and Leon F. Burmeister. “Accuracy of Data in Abstracts of Published Research Articles.” JAMA 281, no. 12 (1999): 1110–1111. https://doi.org/10.1001/jama.281.12.1110
  7. Tang, Liyan, Zhaoyi Sun, Betina Idnay, et al. “Evaluating Large Language Models on Medical Evidence Summarization.” npj Digital Medicine 6 (2023): 158. https://doi.org/10.1038/s41746-023-00896-7
  8. Walters, William H., and Esther Isabelle Wilder. “Fabrication and Errors in the Bibliographic Citations Generated by ChatGPT.” Scientific Reports 13 (2023): 14045. https://doi.org/10.1038/s41598-023-41032-5

Related articles

All articles