Biohacking
ChatGPT answered IVF questions accurately but left gaps doctors flagged
By Life and Health Today Staff, . Life and Health Today.
What the study set out to ask
When patients use a chatbot to research in vitro fertilization before or during treatment, are they getting answers that are complete and safe? That was the question Nicole Fischer, a reproductive endocrinology and infertility fellow at the University of Pittsburgh, brought to the American Society for Reproductive Medicine's annual meeting in Baltimore.
Fischer and her team ran four common questions about the IVF process and its risks through the free version of GPT-5.6 Luna, a large language model (a type of AI that generates text by predicting likely responses based on patterns in its training data). Each prompt was run three times, producing 12 responses in total. Four physicians then rated those responses alongside two documents from the American Society for Reproductive Medicine's ReproductiveFacts.org, scoring each on accuracy, completeness, clarity, safety, and quality on a one-to-five scale.
Where the chatbot fell short
On every measure except clarity, the chatbot's responses came in below the professional society materials. Physician ratings averaged 3.98 for accuracy against a perfect 5.00 for the society documents, 3.40 versus 4.88 for completeness, 2.92 versus 4.88 for safety, and 3.38 versus 5.00 for overall quality. Clarity was the one near-tie: 4.60 for the chatbot, 4.63 for the society materials, according to the analysis reported at the meeting.
The gap in checklist adherence was smaller but still present. The society materials covered 100 percent of key concepts the researchers tracked; the chatbot averaged 94.1 percent.
The more striking finding was in error rates. Half of the chatbot's answers contained at least one unsupported claim. About 4 percent contained misinformation, compared with none in the society documents. Fischer described the pattern plainly: "The key point here is that fluent, confident wording can coexist with omissions, outdated information, and errors."
The team also tested how the chatbot handled a specific factual question: what is the chance of a live birth from IVF at each age from 20 to 45? For most ages, the chatbot's estimates tracked reasonably well against Centers for Disease Control and Prevention and Society for Assisted Reproductive Technology data. It correctly showed birth rates declining with age. But it overestimated live birth rates at 35 and underestimated them at 37 and 40, the analysis found. Fischer's summary of that result: "Topics where accuracy matters most is maybe where these tools are the weakest."
How the question was asked changed the answer
One finding that has direct implications for patients: the way a question was phrased affected the reading level of the response. Conversational prompts produced answers at roughly an eighth-grade reading level; more generic prompts pushed responses closer to a tenth-grade level. The society's own documents sat at about a twelfth-grade level. Fischer read this as evidence that how a patient phrases a question can shape how comprehensible the answer is.
What the study cannot settle
The analysis was small: four physicians, 12 chatbot responses, two society documents, one version of one AI tool. Fischer acknowledged those limits directly, noting that newer and specialized models, including ChatGPT Health, were not tested. Some of the scoring measures were subjective, and the study tracked no patient-level outcomes, meaning it cannot say whether patients who used the chatbot made better or worse decisions as a result.
The study was presented as a conference abstract, which means it has not yet gone through the full peer-review process that a published journal article would require.
What Fischer recommended for clinicians
Fischer's practical advice, directed at clinicians rather than patients, was to ask patients how they are using AI tools, show them how to prompt the chatbot to cite its sources, and point them toward trusted resources. Whether a patient's own use of these tools is appropriate is a conversation that belongs with the clinician who knows their history.
Source: https://www.medpagetoday.com/meetingcoverage/asrm/123668
Life & Health Today provides information for educational purposes only, not medical advice, diagnosis, or treatment. Always consult a qualified healthcare professional.