Uma plataforma de avaliação de Grandes Modelos de Linguagem aplicados à geração de questões educacionais para avaliações diagnósticas de língua portuguesa

Abstract

Assessments play a fundamental role in the educational context, being essential for the construction of student knowledge. Among the different types of assessment, diagnostic assessments stand out, aligned with BNCC skills, which provide prior information about the knowledge of a student or class, allowing educators to adjust their teaching plans and methodologies. Despite its importance, creating educational questions is a laborious process that requires time and experience, especially when the objective is to assess specific student skills. This challenge often makes the process unfeasible for teachers. In view of this difficulty, research in the area of Artificial Intelligence has gained prominence, especially in the field of Question Generation, a branch of Natural Language Processing focused on the automated creation of questions. This research has driven the development of systems based on Large Language Models, which can be applied in the educational context. However, despite the potential of these systems, their adoption still faces challenges, with the quality of the questions generated being one of the main ones. Many focus only on linguistic quality, neglecting pedagogical relevance. This problem occurs mainly due to the lack of tools to assist in assessment and the absence of an established standard in the literature for effective metrics. In view of this, this work proposes a standardized platform for assessing both linguistic and pedagogical aspects of the questions, assisting researchers in validating models for generating educational questions aligned with the BNCC diagnostic assessments. The development of the platform followed a structured process, from requirements elicitation and prototyping to implementation and deployment on free platforms. To validate the platform, we carried out a use case, adjusting the FLAN-T5 model for three Natural Language Processing tasks, allowing them to jointly generate complete and structured educational questions for diagnostic assessments. With the adjusted models, the tool was evaluated through the metrics made available, demonstrating its usefulness and practicality for evaluating the models in the educational context used.