Can ChatGPT pass a physics degree? Making a case for reformation of assessment of undergraduate degrees

Pimbblet, Kevin; Morrell, Lesley

Abstract

The emergence of conversational natural language processing models presents a significant challenge for Higher Education. In this work, we use the entirety of a UK Physics undergraduate (BSc with Honours) degree including all examinations and coursework to test if ChatGPT (GPT-4) can pass a degree. We adopt a "maximal cheating" approach wherein we permit ourselves to modify questions for clarity, split question up into smaller sub-components, expand on answers given – especially for long form written responses, obtaining references, and use of advanced coaching, plug-ins and custom instructions to optimize outputs. In general, there are only certain parts of the degree in question where GPT-4 fails. Explicitly these include compulsory laboratory elements, and the final project which is assessed by a viva. If these were no issue, then GPT-4 would pass with a grade of an upper second class overall. In general, coding tasks are performed exceptionally well, along with simple single-step solution problems. Multiple step problems and longer prose are generally poorer along with interdisciplinary problems. We strongly suggest that there is now a necessity to urgently re-think and revise assessment practice in physics – and other disciplines – due to the existence of AI such as GPT-4. We recommend close scrutiny of assessment tasks: only invigilated in-person examinations, vivas, laboratory skills testing (or "performances" in other disciplines), and presentations are not vulnerable to GPT-4, and urge consideration of how AI can be embedded within the disciplinary context.

Citation

Pimbblet, K., & Morrell, L. (2025). Can ChatGPT pass a physics degree? Making a case for reformation of assessment of undergraduate degrees. European Journal of Physics, 46(1), Article 015702. https://doi.org/10.1088/1361-6404/ad9874

• Journal Article Type: Article
• Acceptance Date: Nov 28, 2024
• Online Publication Date: Nov 28, 2024
• Publication Date: Jan 1, 2025
• Deposit Date: Nov 29, 2024
• Publicly Available Date: Dec 20, 2024
• Journal: European Journal of Physics
• Print ISSN: 0143-0807
• Publisher: European Physical Society
• Peer Reviewed: Peer Reviewed
• Volume: 46
• Issue: 1
• Article Number: 015702
• DOI: https://doi.org/10.1088/1361-6404/ad9874
• Public URL: https://hull-repository.worktribe.com/output/4928963