Hong, J., Peng, C., Tseng, H., & Sung, Y. (2020). Linguistic Feature Analysis of CEFR Labeling Reliability and Validity in Language Textbooks. Journal of Technology and Chinese Language Teaching, 11(1), 57-83.
[洪嘉馡, 彭駿逸, 曾厚強, & 宋曜廷. (2020). 以語言特徵為本的教材分級難度及有效性之驗證. 科技与中文教学 (Journal of Technology and Chinese Language Teaching), 11(1), 57-83.]

Full paper

Abstract/摘要：

Despite the importance of grading language textbooks for teaching and learning, few studies have addressed the issues of reliability, validity, and efficiency of grading texts. This study adopted an automated textbook grading system to examine the grading consistency of five L2 Chinese textbook series labeled with CEFR difficulty levels. Twelve linguistic features were selected to represent the most crucial aspects of text readability: lexicon, semantics, syntax, and cohesion. Both the validity and reliability of grade assignments were tested between and within textbook series. The results suggested that 4 out of the selected 5 textbook series did not assign grade levels accurately reflective of actual text difficulty.

語言教科書的分級不管對於教學或是學習都是非常重要的一環，但是卻很少討論文本分級的可靠性、有效性的研究。本研究以自動分析教科書等級系統，檢測以CEFR作為標示難度的五套華語文教科書的等級一致性。在本研究中，選取了12種不同的語言特徵作為最具分級影響力的關鍵指標，分別取自詞彙類、語義類、語法類及篇章凝聚類等四大語言層面。本研究主要探究在不同教科書之間的分級一致性與相同教科書不同等級的分級，其有效性和可靠性。研究結果顯示，在本研究所選定的五套華語文教科書當中，四套華語文教科書沒有依照實際文本難度進行等級分級。