Weighing it up: Potential benefits and risks of AI assessment

Overview

As discussed in our previous article, Artificial Intelligence (AI) can be used to address various challenges with traditional assessment – and in many different ways. Our project is most focused on developing a learner-focused agent – specifically, an agent that is trained on course materials and existing assessment rubrics to run an oral assessment with a learner and grade them. It is important for us to understand both the benefits and risks of this.

This article is the fourth in a series titled “AI in Education”, aimed at education providers interested in AI. The intention is for this series to act as a beginner’s guide to the use of AI in education, with a particular focus on AI agents. This series is being developed as part of a project to develop an AI agent for learner oral assessment, funded by the Food and Fibre Centre of Vocational Excellence. We invite you to follow along as we (Scarlatti) document our learnings about this exciting space.

The article below provides an overview of the potential benefits and risks of learner-focused agents. We suggest these speculatively, rather than with the intention of being definitive or exhaustive. This is because it is based on industry stakeholder conversations and early desk research, rather than pilots (scheduled for April 2025). We believe the real benefits and risks will become clearer after these pilots.

Note that this article reflects the views of Scarlatti as of February 2025. They do not necessarily represent the views of the Food and Fibre Centre of Vocational Excellence.

Potential benefits

We suggest that an AI agent for oral assessment may provide benefits, such as:

Greater personalisation - In the same way that a human assessor can adapt their approach to suit the learner during an assessment, an AI agent could adapt as the assessment proceeds. This offers the potential for ‘mass-personalisation’ of the assessment process.
Real-time feedback and coaching – Relating to the above, the conversational nature of AI agents means that they could be capable of adapting to learners’ questions and providing personalised feedback. This could help students who are not comfortable asking for support, or who cannot afford an additional tutor.
Improved inclusivity – Learners will provide their answers verbally to the agent. This may benefit learners who have learning difficulties, neurodivergence or English as a second language.
Increased enrolments - People who are deterred from formal study due to the need for written assessments may be more interested in applying to courses that harness these AI agents.
Improved flexibility – As these agents could be accessed remotely (via the internet), learners could have more flexibility to choose when, where and how they carry out assessments. For instance, a student could talk to the agent to undertake an assessment while milking.
Increased digital literacy – Using this technology could aid learners in developing greater digital literacy skills, including familiarity with AI tools.

Potential risks

However, adopting AI assessment agents may also pose risks, such as:

Privacy concerns – Some AI models use the data that users feed into the agent to continue to train the model, which is concerning for learners’ inputting sensitive or personal information. This is a risk that can be managed using different premium subscriptions to the AI model.
Inability for the agent to understand users’ accents – There is also a risk when AI agents are conducting verbal assessments that they could struggle to understand different learners’ accents, leading to issues determining whether they have answered the question correctly and deciding on preliminary grades.
Inaccurate results – Depending on how the agent is designed and what content is inputted into it (i.e. assessment answers, marking rubrics, course content) the agent could misinterpret what the learner is trying to say which could lead to unreliable results. This is particularly concerning for assessments that are set up to only pass learners who mention specific terms in their answers.
Bias in AI models[1] – AI agents could perpetuate biases in the AI model’s training data and could lead them to make discriminatory decisions during the assessment. This is particularly relevant in the context of New Zealand as an AI model originating in the United States is unlikely to have a comprehensive understanding of Mātauranga Māori or Te Reo.
Data exposure[2] – As with any technology, there are also concerns that AI agents could be hacked during assessments or disclose learners’ information without their consent.

[1] We suggest this risk is secondary to the previous ones in this list, given that modern AI models have improved their mitigation of bias. We are currently unaware of any work that demonstrates clear identity-based biases that would be material for our use-case.

[2] We suggest this risk is secondary to the previous ones in this list, given that data exposure risk is no worse than any other cloud-based service.

Scarlatti's take

We believe, depending on the problem at hand, that the potential benefits are enough to warrant at least the exploration of using an AI assessment agent for learners. Within our project, we will explore current guidance, build in mitigations to risks, and collect feedback from different stakeholders.

Questions that we are asking for our own AI agent:

Where can we look for guidance on risks? (e.g., government, thought leaders).
How can we mitigate these risks within our development and pilots?
How will these benefits and risks be realised during pilots and will the benefits be enough?

Interested in following our journey into AI?

Sign up to receive our next article directly to your inbox.
Contact the Scarlatti team to share your thoughts or questions.

Weighing it up: Potential benefits and risks of AI assessment

Phoebe Gill

Overview

Potential benefits

Potential risks

Scarlatti's take

Further reading

Weighing it up: Potential benefits and risks of AI assessment

Phoebe Gill

Overview

Potential benefits

Potential risks

Scarlatti's take

Further reading

You may also like

The global landscape: How AI is transforming education

Shaping the Future: AI education projects in Oceania

Assessment possibilities: Different types of AI for assessment