Assessment possibilities: Different types of AI for assessment

Overview

Artificial Intelligence (AI) presents a promising opportunity to evolve the way that educational assessments take place. Not only could there be benefits for learners and employers, but also for providers (we discuss the benefits specifically of an AI agent for learner assessment in our next article). However, as of early 2025, there are still very few real-world examples of AI agents that undertake assessments in Oceania– and we suspect also worldwide.

This article is the third in a series titled “AI in Education”, aimed at education providers interested in AI. The intention is for this series to act as a beginner’s guide to the use of AI in education, with a particular focus on AI agents. This series is being developed as part of our project to develop an AI agent for learner oral assessment, funded by the Food and Fibre Centre of Vocational Excellence. We invite you to follow along as we (Scarlatti) document our learnings about this exciting space.

The article below provides a brief overview of AI for assessments – the form this can take, what features an AI agent for assessment could have and how we envision different stakeholders using them. It is important to note that there are no pre-existing categorisations or definitions of what an AI assessment agent is for us to draw from. The below is therefore a conceptual piece based on our own thinking.

What forms of solution exist?

Before getting into AI for assessment – it may be worthwhile understanding the forms of solution that could exist for it. As mentioned in a previous article, AI can be used for assessment to generate tests, administer them, grade students, or interpret insights to provide additional support to users. However, the exact solutions used in different instances may not necessarily be a ‘chatbot’. Our impression is that some terms are currently being used interchangeably. Below, we offer an attempt at definitions:

AI Platform

An AI platform is an application that brings together a range of AI services in one place. Through these platforms, users can build, deploy, and manage AI tools, chatbots and agents. Examples of AI platforms include Microsoft Azure AI, Google Cloud Platform, or the University of Sydney’s Cogniti platform, which is specifically for educators to develop their own agents (see more about this in our earlier article).

AI Tool

AI-powered tools are tools designed to perform a specific function or set of tasks using AI algorithms. It will often involve a user interface with buttons, menus, or other forms of structured input. In some cases, it may simulate a human-like conversation with the user, for example, a quiz generator.

Chatbot

A generic pre-large language model chatbot performs pre-defined functions and answers common questions. Unlike advanced AI systems (e.g., ChatGPT), pre-LLM chatbots are often rule-based or use natural language processing to generate structured responses, for example, a website’s ‘help desk’ chatbot.

AI Agent

By comparison, AI agents (confusingly also being an AI chatbot by many in early 2025) are designed to simulate human-like conversations and handle multiple tasks. Agents do so by focusing on real-time interactions and responding dynamically to questions or prompts. It can provide information, perform actions, or guide users interactively, for example, Siri by Apple or Alexa by Amazon.

What types of AI for assessment could exist?

Our desk research has shown that AI for assessment has the potential to mean many things. For example, it could vary by:

Problem – It could address a range of challenges with assessments
Assessment stage – It could undertake different stages of the assessment process
Solution type – It could be a tool, chatbot or agent
User – It could be used by different people (e.g., a learner, a provider, an employer)
Speech or text-based – It could talk to the user via speech or text
Question type – It could involve open-ended or closed questions
Training type - It could be used in formal or non-formal education.

Below we show examples of New Zealand organisations currently exploring AI for assessment, including our best guess of the characteristics of what they are developing.

What features might an AI assessment agent for learners have?

With the support of the Food and Fibre Centre of Vocational Excellence, Scarlatti (the last row in the table above) is aiming to develop an AI agent for oral assessment of VET learners. Below, we explore features we think such an agent could have. We have divided these into ‘must-haves’ and ‘nice-to-haves’. Within our project we will undertake all ‘must-haves’ and some of the ‘nice-to-haves’.

Must-haves

We imagine that to be used in assessment, our AI agent would at least need the following capabilities (Clark, 2024):

Contextual knowledge base – The agent would need its own database based on course materials, assessment rubrics, and/or model answers.
Human-like conversation – To replicate human interactions, the agent would need to have:
Adaptability – It should dynamically adjust the difficulty of assessments and type of questions based on learners’ responses.
Verbal communication options – The agent would have the option of verbally communicating with the user and responding to their answers in real time. Ideally, this would use a natural-sounding voice with an embedded accent such as Amazon’s AWS Aria in the context of New Zealand.
Voice recognition and transcription – The agent would also be able to record the learner’s verbal answers and transcribe them into text, using existing tools such as OpenAI’s Whisper model.
Grading – Evaluate responses against predefined answers and provide consistent, accurate grading.
Ethical framework – A contextually appropriate framework (e.g., appropriate for New Zealand, the time period in which the agent is being built, and the users it will engage with) upon which the AI agent is built and piloted.

Nice-to-haves

Depending on the needs of the learners, including the formality of assessment and sensitivity of the topic, some other features that we could add include:

Authentication - Authenticate assessments using a secure login or identity verification.
Preamble – Explanation upfront that details the purpose of the assessment, how the AI agent functions, how performance is evaluated, and where one can go with concerns.
Scenarios – Verbal scenarios and simulations that mimic real-world situations.
Personalisation – Personalised interactions based on the student’s history, learning objectives, and preferred focus areas.
User response and assessment result storage – The storing of audio recordings, transcripts, grading, and/or qualitative feedback.
Progress tracking – It would be able to measure the learner's progress in their assessments and studies.
Qualitative feedback – Provides the learner with feedback on how they can improve and in what areas.
Report generation – It could produce a report that summarises the conversation with the learner.
Gamification – Incorporating elements of games, point systems, and rewards to make it more interactive for learners.

Note: Only some of these features would require AI.

Scarlatti's take

We believe that AI may provide a significant opportunity to positively redefine assessments in Aotearoa. Desk research has shown us that “AI for assessment” is not as simple as an agent that runs oral assessments with learners and grades them – as we originally hoped might be the case. In fact, AI has the potential to solve many challenges with traditional assessment, to the benefit of stakeholders like learners, providers, and employers.

Questions that we are asking for our own AI agent:

What are the specific benefits and risks of an AI agent for oral assessment with learners?
How can we maximise and mitigate these, respectively?
How will these benefits and risks be realised during pilots and will the benefits be enough?

Interested in following our journey into AI?

Sign up to receive our next article directly to your inbox.
Contact the Scarlatti team to share your thoughts or questions.

References

Piven, I. (2024). QS Reimagine Education (Best use of Generative AI). Retrieved January 10, 2025, from QS Reimagine Education (Best use of Generative AI)

Clark, D. (2024). Taming the AI and Assessment Conflict: 21 Practical Resolutions. Retrieved January 10, 2025, from https://donaldclarkplanb.blogspot.com/2024/10/taming-ai-assessment-conflict-15.html

ConCOVE. (2025). AI Generated Assessment. Retrieved January 10, 2025 from https://concove.ac.nz/concove-projects/ai-generated-assessment/

Gerritsen, J. (2024). Could AI be used to help write school exams? NZQA thinks so. Retrieved January 10, 2025, from https://www.rnz.co.nz/news/national/515688/could-ai-be-used-to-help-write-school-exams-nzqa-thinks-so