Poole, F. J., & & Coss, M. (2024). Can ChatGPT Reliably and Accurately Apply a Rubric to L2 Writing Assessments? The Devil is in the Prompt(s). Journal of Technology and Chinese Language Teaching, 15(1), 1-24.

Full paper

Abstract/摘要：

This paper investigates the effectiveness of ChatGPT, a generative AI tool, in assessing second language (L2) writing. The study explores the practicality of employing ChatGPT as an assessment tool, focusing on the accuracy and reliability of the AI-generated scores compared to human raters. Various prompting strategies were tested to understand their impact on the effectiveness of ChatGPT in this context. The paper also examines the reliability of ChatGPT scores across different writing topics. The findings demonstrate that ChatGPT can serve as a valuable tool in L2 writing assessment, provided that it is used strategically with well-crafted prompts. The study contributes to the growing body of research on automated writing assessment tools, particularly in the realm of L2 learning, and offers insights into the practical application of such tools in educational settings.

本文研究了ChatGPT在评估二语（L2）写作中的有效性。研究探讨了使用ChatGPT作为评估工具的实用性，重点关注AI生成的评分与人工评分相比的准确性和可靠性。为了理解不同指令策略对ChatGPT在这一背景下有效性的影响，本研究测试了十种指令策略。本文还检验了ChatGPT在不同写作主题上的评分可靠性。研究结果表明，ChatGPT可以作为L2写作评估中的一个有价值的工具，前提是要使用精心设计的指令策略。本研究为自动写作评估工具的研究提供了新的见解，特别是在L2学习领域，并提供了这些工具在教育环境中实际应用的见解。