Aller au contenu

Evaluation

Ce contenu n’est pas encore disponible dans votre langue.

Tabiya evaluates Compass at four levels. The levels follow the AI Evaluation Playbook, a framework for evaluating generative AI tools in global development. The Agency Fund, IDinsight, and the Center for Global Development maintain it. Each level answers a different question.

Level Question Evaluation of Compass
1. Model Does the AI system perform as intended? Eight kinds of tests, from synthetic queries to a manual review of more than 1,000 pilot conversations
2. Product Does the overall product engage and retain users? Moderated user tests in South Africa in 2024, and a test at scale with jobseekers in Kenya from April 2025
3. User Does the product change users’ thoughts, feelings, knowledge, and behavior toward the development outcome? Measured within the randomized controlled trials, with results pending
4. Impact Do users with access to the product improve development outcomes? Randomized controlled trials in South Africa, Kenya, and Argentina, with results pending

Tabiya moved to an impact evaluation after the product was stable and usable. Evaluation in the Compass Connect section describes how the four levels apply to a deployment of Compass Connect.

Model evaluation checks that Compass extracts accurate information, stays on topic, and produces output that jobseekers can use. Language models can state things that are wrong, including skills that a person never mentioned. Tabiya added tests step by step as the tasks of the system became clear. There are eight kinds.

Test What it checks
Embedding evaluation Which properties of a taxonomy entry to include in its embedding, and how many candidates the search should return. The tests use established datasets and synthetic queries.
Synthetic queries Whether the search finds the right occupation or skill in the taxonomy for a description in everyday language. The test set has 542 occupation queries and 1,054 skill queries. The measure is recall.
Isolated components Whether each tool returns the expected output for a known input. One tool, for example, classifies an experience as paid work, volunteering, or a personal project.
Summarization Whether the summary of the earlier conversation is stable, keeps the key details, and is faithful to what was said.
Scripted dialogues Whether each agent collects the right information and hands over correctly to the next agent.
Simulated users Whether whole conversations work when a test driver plays users with different personas. The personas come from user research.
Wizard of Oz test Whether the conversation design works for users. In a small study on WhatsApp in Kenya, a developer wrote Compass’s replies by hand.
Pilot review Whether the system behaves as intended with users. The team reviewed more than 1,000 pilot conversations by hand and traced the identified skills back to what users had written.

Users can also flag a message in Compass, for example as offensive, biased, or inaccurate.

The measures depend on the test: recall for the taxonomy search, F1 scores for classification, and task completion, conciseness, and relevance for conversations. For judgments such as relevance, human reviewers and a second language model both assess the output.

The test code and a guide to running it are in the Compass repository. The Technical Overview lists the tests from a developer’s point of view. The blog post Building Responsible AI for Jobseekers: Model Evaluation of Compass (October 2025) describes seven of the tests in detail.

An accurate model is not enough. People also have to be able to finish the conversation, understand what Compass asks, and find the result useful. Tabiya tested this in two ways: in moderated sessions with a few jobseekers, and at scale.

Moderated User Tests in South Africa, August 2024

Section titled “Moderated User Tests in South Africa, August 2024”

With Harambee Youth Employment Accelerator, Tabiya ran three days of user tests in Cape Town in August 2024, on an early version of Compass. Nine jobseekers between the ages of 18 and 35 took part. Each used Compass on a smartphone while a moderator from Harambee observed, and answered questions afterward.

What participants said:

  • Eight of nine found Compass easy to use and its responses easy to understand.
  • Seven of nine found the skills that Compass identified accurate and relevant.
  • Eight of nine said Compass helped them understand their experiences and skills better.
  • All participants who completed the test said they would recommend Compass to other jobseekers.

What the sessions showed:

  • Compass brought the conversation back on topic when a participant misunderstood a question, and it handled most spelling mistakes.
  • The conversation took longer than planned. Only one participant finished it within the session.
  • Some questions were misunderstood. Asked what a typical day at work was like, one participant described the atmosphere at work and not the tasks.
  • Compass was available in English only. At times, the moderator explained things to a participant in the participant’s first language.
  • Questions about unpaid work caused more errors than questions about paid work. Some experiences appeared twice in the summary, and activities that happen from time to time did not fit a format with a start date and an end date.

Nine participants are too few to generalize from. The tests showed where the conversation needed work. Later versions of Compass added more languages, editing of experiences and skills, and CV upload.

From April 2025, Tabiya tested Compass with Ajira Digital, a digital jobs program backed by the Kenyan government. The test had two aims. The first was to check that login, sessions, and the system as a whole worked with many users at once. The second was to check that onboarding and the conversation worked for jobseekers.

In the earliest phases of the test, more than 90 percent of users dropped out before the end. Poor connectivity played a part. The data also pointed to four problems with the product:

  • Login and session handling failed when many users arrived at once, and users lost their progress.
  • Users did not know how long the conversation would take.
  • Users did not understand what Compass was asking or why.
  • The questions about skills felt repetitive. The more experiences a user described, the more likely the user was to stop.

The phases of the test ran one after another and were not randomized. After each phase, Tabiya made changes and measured again:

  • Sessions reconnect, and users keep their progress.
  • Users can start without a registration code.
  • A progress bar and time estimates show how much remains.
  • Short onboarding messages explain what Compass will ask and why.
  • Prompts are shorter, and long sequences are split into steps.
  • Fixed rules replace decisions of the language model at some transitions. For example, Compass collects at least one complete experience before it moves on to skills.

By the final phase, 44 percent of users dropped out.

One change showed that a product decision can alter the behavior of the model. Compass first asked about volunteering last. For many users, the most relevant experience was informal or voluntary, and they left before they reached that question. When volunteering moved to the front, Compass treated users as if they had no experience, because its logic had used the volunteering question as a fallback. Tabiya removed the logic that depended on the position of a question.

Platform data, short surveys within the conversation, and follow-up conversations with users added four lessons:

  • Users prefer structured guidance that explains what the tool will produce to an open-ended chat.
  • Shorter prompts, taken one step at a time, lead to more complete answers.
  • Wording matters for unpaid work and care work. Examples such as household management, informal sales, or community leadership made these experiences easier to report.
  • Users are more likely to keep a skill when Compass shows which of their statements it came from.

The blog post From Prototype to Product: Testing Compass with Kenyan Youth (January 2026) describes the test in detail.

Tabiya measures changes in jobseekers’ beliefs, confidence, and job search behavior as intermediate outcomes within the randomized controlled trials described under level 4. Results are pending.

Randomized controlled trials of Compass are ongoing in South Africa, Kenya, and Argentina. Results are pending. The pre-analysis plan for the trial in South Africa is in the AEA RCT Registry.