Avaliação da qualidade de testes python gerados por LLMs

Abstract

The manual generation of test scripts is a time-intensive, costly, and error-prone process, emphasizing the importance of automated solutions. Large Language Models (LLMs) have demonstrated significant potential in this area by leveraging extensive knowledge to generate test code more efficiently. This study examines the quality of Python test code produced by three LLMs: GPT-4o, Amazon Q, and LLama 3.3. The structural reliability of test suites is evaluated under two distinct prompt contexts: Text2Code (T2C) and Code2Code (C2C). The analysis involves identifying errors and test smells, with a particular focus on their correlation to inadequate design patterns. The findings indicate that most test suites generated by the LLMs contained at least one error or test smell. Assertion errors were the most prevalent, accounting for 64% of all identified errors, while Lack of Cohesion of Test Cases was the most frequently detected test smell (41%). Prompt context played a significant role in test quality, as textual prompts with detailed instructions tended to generate tests with fewer errors but a higher incidence of test smells. Among the evaluated LLMs, GPT-4o produced the fewest errors in both contexts (10% in C2C and 6% in T2C), whereas Amazon Q exhibited the highest error rates (19% in C2C and 28% in T2C). Regarding test smells, Amazon Q had fewer detections in the C2C context (9%), while LLama 3.3 showed better performance in the T2C context (10%). Additionally, a strong relationship was observed between specific errors, such as assertion and indentation issues, and test case cohesion smells. These results highlight opportunities to enhance the quality of LLM-generated tests and emphasize the need for further research into optimized generation scenarios and improved prompt engineering strategies.