Evaluating Nepali Sexual, Reproductive and Maternal Health Chatbot with Large Language Models (LLMs)

Bishesh Khanal of the Nepal Applied Mathematics and Informatics Institute for Research in Nepal will assess LLMs for their ability to provide accurate information on sexual, reproductive, and maternal health (SRMH) topics in Nepali to the general public and female community health volunteers. In Nepal, limited access to SRMH resources due to language barriers and social stigmas has led to increased numbers of unsafe pregnancies and sexually transmitted diseases. While LLMs could be helpful, they have many limitations, particularly in low-resource, non-Western settings. These include inaccurate responses, poor performance in non-English languages, responses generated largely from Western-cultural contexts, and large computational resource requirements. Together with a local multidisciplinary team, involving AI scientists, domain experts, and community engagement experts, they will integrate four chatbots into a simple mobile-friendly web-interface, and evaluate their performance to anonymous chat queries from 5,000 individuals.