Methodology
How our Scoring and Feedback Work
VirtualSpeech helps learners practise conversations, understand their performance and decide what to try differently. This page sets out how a roleplay is assessed, what we measure and where the limits are.
Scope. This page explains the standard roleplay assessment process. Presentations, focused exercises and standalone coaching sessions can use different assessment methods.
Step 1
Criteria that fit the conversation
After a roleplay, AI assesses the learner's contributions against scenario-specific criteria. Each standard roleplay has five assessment criteria, with descriptions explaining what each criterion covers.
For example, a difficult conversation might assess:
- Asking open-ended questions
- Acknowledging concerns
- Using non-confrontational language
Other scenarios use criteria suited to their objectives. The scoring system receives:
- CriteriaThe names and descriptions of the five criteria
- RolesThe learner's role and the avatar's role
- TranscriptThe full conversation transcript
Step 2
Assessment based on the learner's contributions
The AI is instructed to assess what the learner said, using the avatar's responses to understand the context.
It must distinguish between an idea the learner introduced and an idea supplied by the avatar. Where a criterion concerns responding appropriately, the learner can receive credit for acknowledging or building on something the avatar said.
The system is instructed to support positive assessments with evidence from the learner's own contributions and to avoid inventing examples.
Keep in mind: These content assessments use the transcript. They do not directly measure vocal tone, facial expression or body language.
Step 3
How the scores are calculated
For a standard roleplay, the system requests three separate AI assessments of the same transcript against the same criteria. These are repeated assessments, not assessments by three different human experts.
Each assessment returns a whole-number score from 0 to 10 for each criterion. The system averages the available successfully returned scores for each criterion and rounds the result to one decimal place.
Worked example for one criterion
Scores of 7, 8 and 8 produce a criterion score of 7.7.
The content score gives equal weight to the five criteria, subject to rounding.
Keep in mind: Repeated assessment is intended to reduce dependence on a single AI response. It does not establish accuracy or eliminate variation.
Step 4
Feedback that explains what to practise next
One of the three assessments also generates written feedback for each criterion. The AI is instructed to explain the learner's strengths, identify improvements and use specific examples from the conversation.
The displayed criterion score is an average across assessments. Its written explanation comes from the assessment that generates feedback.
A separate feedback summary uses the scenario instructions, rubric and transcript to identify strengths, areas for improvement and practical takeaways. Where there is too little relevant participation to give meaningful feedback, it is instructed to explain that limitation rather than invent strengths.
Alongside the content assessment
Communication metrics
The platform also calculates indicators that help learners reflect on their delivery.
| Metric | How it is calculated | Where |
|---|---|---|
| Speaking pace | Words per minute, based on word count and recorded speaking time. | All sessions |
| Detected filler words | Matches to configured words and phrases in the transcript. | All sessions |
| Listenability | A text-based estimate using sentence length and syllable counts through the Flesch Reading Ease formula. | All sessions |
| Looking towards the avatar | An estimate based on headset viewing direction. | Supported VR roleplays |
| Movement | An indicator derived from recorded movement. | Supported VR sessions |
Keep in mind: Transcription affects word counts and filler detection. Text readability does not directly measure pronunciation or vocal clarity. Headset direction estimates where someone is looking rather than measuring precise eye gaze.
Putting it together
The overall score
The overall score combines the content assessment with the delivery indicators. The weighting depends on the type of practice.
Where a delivery component is not measured, for example looking towards the avatar and movement in browser sessions, it is not included in the calculation.
After practice
Turning feedback into reflection
After practice, learners can discuss the session with the AI coach.
The coach receives the scenario context, transcript, scores and feedback. It is instructed to help learners reflect on their performance, revisit specific moments, practise alternative wording and identify actions for a real conversation.
Learners are encouraged to consider the feedback alongside their own judgement.
Reading the numbers
Understanding progress
Scores describe performance in a particular simulated session. Comparisons are most useful when the scenario, criteria, difficulty and device setup are comparable.
A higher practice score is evidence of stronger performance against that assessment. It does not, by itself, demonstrate better workplace performance.
Keep in mind: AI assessments can vary or make mistakes. Scores should support learning, reflection and discussion alongside other evidence.
External review
We are carrying out an initial review with independent expert reviewers, who compare their independent assessments with our AI scores. This runs while we prepare a longer study with university researchers.
Once completed, we will publish the scope, method, findings and limitations here. No results from the planned review are available yet.
Independent research using the platform is listed on our research page.