Fine-tuning improved automated discharge-summary scores in a 50-patient evaluation, while adding retrieval did not yield a measurable improvement.
More components did not guarantee better performance.
What researchers studied
Researchers compared three open-source language models generating discharge summaries from intensive-care records relevant to post-surgical cancer care.
Using MIMIC-IV data, they evaluated pretrained and parameter-efficient fine-tuned versions of each model, with and without a hybrid retrieval pipeline.
This produced 12 configurations tested on the same fixed set of 50 patients.
Generated summaries were compared with clinician-authored references using ROUGE, BLEU and BERTScore, alongside paired statistical comparisons and bootstrap confidence intervals.
The work evaluated text generation rather than patient outcomes.
What they found
Fine-tuning improved all three models on every reported metric, with confidence intervals for the gains excluding zero.
The smallest model, StableLM-Zephyr-3B, achieved the highest ROUGE-1 score after fine-tuning, at 0.278.
Retrieval augmentation did not produce a measurable benefit in these configurations.
Importantly, the retrieval method supplied style examples from training cases rather than each patient’s complete clinical evidence.
The result therefore does not show that all retrieval-augmented medical AI is ineffective.
What it means
The comparison suggests healthcare AI teams should test carefully tuned baselines before assuming that retrieval infrastructure will improve clinical documentation.
Smaller models may offer computational advantages, but text similarity does not establish medical safety.
Discharge summaries must accurately communicate diagnoses, treatment, complications and follow-up plans.
Human oversight and direct checks for factual omissions and harmful statements remain necessary before any clinical use.
This study does not demonstrate reduced clinician workload or improved care.
Important limitation
The evaluation involved only 50 patients, a specific dataset, selected models and a particular retrieval approach.
Automated overlap metrics may reward similar wording while missing clinically important errors.
Some information was derived from notes, adding constraints to interpretation.
The findings do not establish that the summaries are ready for unsupervised use or that retrieval generally offers no value in healthcare.
Source Information
Study: Retrieval-augmented Large Language Models for automated discharge summary generation in post-surgical ICU cancer patients
Authors: Sanjib Raj Pandey; Zubeen Khalid; Mahdi Maktab Dar Oghaz; Vin Paleri; Vishak Muraleedharan Shylaja
Journal / Source: Frontiers in Artificial Intelligence
Published: 10 November 2026
