Abstract

Since the release of ChatGPT, researchers have shown growing interest in using large language models (LLMs) for academic writing. This study explored the possibilities and limitations of AI use in summarizing the core content of research papers into abstracts. We analyzed 204 papers published in the Journal of the Korean Society for Library and Information Science (2022-2024) by comparing author-written abstracts with those generated by ChatGPT. Quantitative analyses examined similarities among the originals, AI-generated abstracts, full texts, and different prompt types, supplemented by expert perception. Results showed that AI-generated abstracts were semantically close to the originals based on BERT and TF-IDF scores but differed in word choice and expression. Abstracts generated with Korean prompts showed the highest similarity to both the originals and the full texts, indicating that the prompt language affected style and content representation. Experts viewed LLMs as helpful tools for improving clarity and fluency in writing. Overall, the findings suggest the potential of LLMs as collaborative partners in abstract writing.