Use of Foundational AI in teaching and assessment – A practical case study in software engineering education

Project Description

The emerging foundational AI systems such as ChatGPT and Bard, and software focused tools such as Github's Copilot are creating a challenge for how we teach and practice software development. Based on these challenges we propose some learning and teaching strategies to keep pace with AI. For example, from a learning perspective, we can encourage students to use AI in pair programming where one member of the pair is an AI. This requires students to show the evidence of the questioning and critical evaluation process including explaining how to identify and scientifically select the replies given by AI. Showing an understanding of how to verify the correctness and rationality of the replies can be a critical task, especially given that current foundational AI tools are known to produce some erroneous or biased replies. From an assessment perspective, controlled digital assessments based on tracking technologies combined with AI can be used to identify how students interact with their devices. AI may even provide us with opportunities to identify and flag text that may have been generated by AI or unusual patterns of behaviour that may indicate cheating, such as long pauses when answering questions and sudden bursts of high-quality writing. The aim of this project is to verify our proposed strategies in the form of a case study and to summarize a set of general teaching design and assessment recommendations, enabling the effective use of foundational AI systems in educational settings.

The main elements of the project include designing assignments or questions when using foundational AI systems as a learning assistant and designing scientific evaluation methods for assignments using AI replies. To verify the generality of the method, two different representative courses of software engineering will be selected. One is a critical thinking module that does not demand too advanced programming skills but is crucial for software development. The other is a system development module, which requires good programming skills. The target students are also divided into two types: students who are new to programming in their first year and students who already have some programming skills in their second or third year.

The implementation of the project will produce results in two dimensions. On the one hand, from the perspective of students, it is expected to produce a set of assistive learning methods that regulate and effectively use foundational AI tools. On the other hand, from the perspective of teachers, we hope to summarize a set of teaching design and assessment guidelines based on the use of foundational AI tools, so that teachers can effectively use and incorporate those tools to generate good assessments and engaging content that can better inspire students to learn.