AI Architecture

Compass utilizes agentic workflows to interact with users, gather information, and identify their skills.

Compass mimics how a human would approach a conversation and the resulting tasks, acting as an overarching agent that decomposes into smaller agents, each with its own responsibilities and goals.

Each agent within Compass has a specific responsibility and performs multiple tasks to achieve its goal. For example, an agent might converse with a user to collect specific information, process that information, and prepare it for use by another agent.

Agents maintain an internal state that allows them to apply a strategy to accomplish their goal, and have access to the user's conversation history and use tools based on LLM prompts (or not). These tools could be used for tasks such as conversing with the user, named entity extraction, classification, or transforming user input.

Agents are guided by a combination of instructions (prompts) and their internal state. The prompts can vary based on the agent's state to help them accomplish their tasks.

Once Compass has gathered all the necessary information from the user, it processes the data to identify the user’s top skills. This identification is done through a multi-stage pipeline that employs various techniques, such as clustering, classification, and entity linking the a occupations/skills taxonomy.

Compass leverages LLMs in four ways:

1. Conversational Engagement: Unlike typical applications where the LLM responds to user questions, Compass reverses this interaction. It generates questions to guide a directed and grounded conversation.

2. Natural Language Processing Tasks: Compass uses the LLM for tasks like clustering, named entity extraction, and classification, handling user inputs efficiently without the need for costly and time-consuming model training or fine-tuning.

3. Explainability and Traceability: The LLM provides reasoning for specific outputs, allowing for explanations that link discovered skills back to the user’s input. This feature, which is based on a variation of Chain of Thought reasoning, is especially noteworthy—not only for the capability it offers but also because it was an unplanned outcome. It emerged while attempting to solve a different problem: improving the accuracy of the LLM's tasks

4. Filtering of Taxonomy output: The LLM filters relevant skills and occupations connected to the ESCO model from the conversation's output. Leveraging its advanced reasoning capabilities, the LLM efficiently processes large amounts of text, identifying the most pertinent entities. This approach combines traditional entity linking via semantic search with LLM-based filtering, creating a hybrid solution.

Compass is grounded and protected from hallucinations in multiple ways:

• Task Decomposition: Smaller, more manageable tasks are assigned to individual agents with specific LLM prompts.

• State-Induced Instructions: Agents use their internal state to generate targeted instructions during user interactions, guiding the conversation toward a specific goal. This approach reduces the size of the prompt by including only relevant segments, making it more likely that the LLM will follow the instructions accurately, thereby reducing the risk of hallucinations.

• Guided Output: Instructions are carefully crafted to increase the likelihood of relevant responses. Techniques include:

  • One- and few-shot learning

  • Chain of Thought

  • Retrieval Augmented Generation

  • JSON schemas with validation and retries

  • Ordering output segments to align with semantic dependencies

• State Guardrails: Simple, rule-based decisions are made whenever possible, reducing reliance on the LLM and minimizing potential inaccuracies.

• Taxonomy Grounding: By linking entities to a predefined occupations/skills taxonomy, Compass ensures that identified skills remain within a relevant and accurate domain.

Evaluation

For evaluating Compass, we followed the strategy outlined below:

• Rigorous Embeddings Evaluation: We rigorously evaluated various strategies for generating embeddings from the taxonomy entities. Our considerations included identifying which properties of the entities should be included in the embeddings generation, as well as determining the optimal number of entities to balance accuracy and precision. For the tests, we used established datasets from the literature and generated synthetic data to mimic Compass user queries.

• Isolated Component Testing: Each agent's tools were evaluated individually using specific inputs and expected outputs. For example, classification components were tested with known inputs to verify accurate label assignments.

• Scripted Conversations: Conversational agents were tested using predefined dialogues, with outputs evaluated by either automated evaluators (other LLMs) or human inspectors.

• Simulated User Interactions: Compass was tested in end-to-end scenarios by simulating user interactions driven by an LLM. The simulated user was given a persona based on our UX research and additional instructions to cover specific cases of interest. These conversations were then assessed for quality and relevance by automated evaluators (other LLMs) or human inspectors.

• User Testing and Trace Analysis: On a smaller scale, real user tests were conducted. By tracing the top skill outputs back to the user's input, human inspectors could assess the performance of specific agents within Compass.

The Core Role of a Taxonomy

Tabiya's inclusive taxonomy plays a central role in Compass. It grounds the LLM’s tasks, but there are several additional aspects worth mentioning:

• Standardization: The identified skills are linked to a standard taxonomy, making interpretation and comparison easier. The concepts behind these skills are well-defined and can be explained, allowing for clarification and disambiguation.

• Canonicalization: Explored skills are listed with canonical names and UUIDs, enabling consistent referencing across different experiences and applications.

• Network Structure: The taxonomy models the labor market by associating occupations with skills, forming a knowledge graph that can provide additional insights to users.

• Unseen Economy: The taxonomy has been extended to include activities from the unseen economy, empowering young women and first-time job seekers to enter the job market.

• Localization: A taxonomy can consider the specific context of a country. This includes occupations unique to certain regions, alternative names for occupations that are region-specific, and varying skill requirements for the same occupation across different countries.

• Work Type Classification: All experiences are classified into four types (wage employment, self-employment, unpaid trainee work, and unseen work), which allows for a more targeted exploration of the job seeker’s skills.

Technical Stack Overview

• Language Models and Embeddings: Compass utilizes the gemini-1.5-flash-001 model for its LLM capabilities and the textembedding-gecko version 3 model for embeddings. The gemini-1.5-pro-preview-0409 model is used for the LLM auto-evaluator. The Gemini model was chosen for its balanced performance across task accuracy, inference speed, rate availability, and cost.

• Backend Technologies: Developed with Python 3.11, FastAPI 0.111, and Pydantic 2.7 for a performant server-side environment. An asynchronous framework suited the use case well, as LLM inference endpoints can be slow. Python was chosen for its extensive AI/ML library support and because it made it easier to integrate ML scientists into the development team.

• Frontend Technologies: The UI, built with React.js 19, TypeScript 5, and Material UI 5, is optimized for mobile but performs well on tablets and desktops. Additionally, we use Storybook 8.1 to showcase, visually inspect, and test UI components in isolation.

• Data Persistence: Data is securely stored using MongoDB Atlas, which includes vector search capabilities. Our team was already familiar with MongoDB, and the taxonomy was already in MongoDB Atlas, so it was a natural choice.

• Deployment: The entire application is deployed on Google Cloud Platform (GCP), ensuring high availability and scalability. We use Pulumi to deploy nearly all the infrastructure, as it allows us to write deployment code in Python, aligning with the rest of the backend development. Additionally, our team was already experienced with Pulumi, making it a natural choice. For error tracking and application performance monitoring, we use Sentry.

Background and Motivation

Tabiya, in partnership with Harambee Youth Employment Accelerator, conducted a three-day series of user experience (UX) tests devised to answer the following questions:

1. Can participants initiate and maintain a natural conversation with Compass?

2. Does Compass effectively guide users through the skill identification process?

3. Do users find the skills identified by Compass relevant and accurate?

4. Are participants satisfied with the overall experience and feel they have a clearer understanding of their skills?

In the South African case, the envisaged users of Compass are the country’s youth, who face high and rising disengagement in the labor market, education, and training rates. This is due, in part, to job search distortions. That is, work-seekers may have incomplete information about their skills. This means that work-seekers could apply for jobs that they are incompatible with or stop searching altogether. Compass is built to mitigate this particular search distortion, which forms part of the drivers of unemployment, especially amongst the youth.

Methodology

South African youth (between the ages of 18 and 35) subscribed to Harambee’s SA Youth Mobi Platform were shown an advertisement to join a UX testing session hosted at Harambee offices in Cape Town. Upon receipt, 604 were deemed most compatible with the selection criteria. This criterion includes the following:

1. Young person is subscribed to SA Youth Mobi Platform

2. Young person ordinarily resides no more than roughly 15 kilometeres away from the Harambee Cape Town office

3. Young person has had work experience before

4. Young person is actively looking for employment

From this criterion, black female participants and participants who had evidence of starting their own small business were favored in the final UX test participant selection.

Before the UX test session begins, each participant is given a consent form to read and sign. Where applicable, participants were asked to sign a photography consent form if they were comfortable doing so. Following this, each UX test comprised four parts. The first part consisted of an introduction and asking the participant for permission to record the session. This part of the UX testing session was also used to explain what Compass is and how it works to the participant.

The second part consisted of the commencement of the UX test. During this part, each participant interacted with Compass as independently as possible until their experiences and skills were summarized, or until the moderator indicated they should stop interacting with Compass due to a time constraint. The UX test involved Compass gathering information from participants on their work experiences, including paid and unpaid work in establishments or non-establishment settings. This is referred to as the work experience section throughout this report. Following this, Compass produced an experience summary listing the participants' experiences, locations, and durations chronologically. Next, Compass gathered information on what skills participants had obtained from each work experience through an interactive dialogue. Up to five skills per experience were subsequently listed at the end of the dialogue. This is referred to as the skills assignment section throughout this report.

During the third part of the session, the moderator asked the participant a series of post-test feedback questions. Finally, the moderator closed the session by giving each participant the timeframe to expect their monetary incentive (of R450) for participating in the UX test. The closing was also used to ask participants if they were comfortable with being placed in a Harambee WhatsApp group for an expert panel that is periodically invited to participate in Harambee sessions targeted at its youth database. All four parts were projected to take 45 minutes but could take up to one hour. In addition to R450, each participant was given a snack pack before each UX test. This is done to ensure that participants had food to eat before commencing their UX tests. The complete discussion guide for each UX test can be found here.

Compass was designed to extract information from its users conversationally. As such, Compass functions similarly to a messaging application. Compass textually asks its users for information pertaining to their experiences and allows users to respond before concluding or probing for more information. Given that this exercise operates similarly to an interaction where someone would communicate through text messaging on their smartphone, the UX test was conducted on a cellphone. For heightened accessibility and ease of use, the cellphone chosen was the Samsung Galaxy A23 because this is the most widely used smartphone among the youth in Harambee’s database.

Results

1. Demographics

A set of 11 SA Youth subscribed youth were selected to test the functionality of Compass. Three of these participants tested out Compass on the 27th of August. Two were male, and one was female. Four were scheduled to complete the testing on the 28th, but there was an attrition rate of 50% on this day. A male and a female showed up; the two not in attendance were male. The 29th of August was the final day of testing. Four participants, three female and one male, each tested Compass. Therefore, 82% (9 people) of the chosen cohort of 11 people participated in the UX Testing. 44% of the UX testers participants were male and 66% female.

1. Timing

From the nine young people who tested Compass, two groups emerged. Group 1 comprised three participants who received a work experience summary but not a skills summary. Group 2 included six participants who received both work experience and skills summary. Group 1 took an average of 32 minutes to complete the entire UX testing session. Group 2 took an average of 32 minutes and 3o seconds to complete. Overviews of each UX testing session, including details on the timings of each session can be found here.

The full Compass experience was completed without interruption only once. This was done by Participant 9 who only possessed one experience and thus, one set of skills. Hence, the initially projected time of 45 minutes to complete the entire UX testing session was an underestimation. Instead, all participants finished within 45 minutes only because the moderator of the participant sessions prematurely instructed participants to finish their interactions with Compass. Given that participants were paid for their time, not doing this would have violated labor laws.

1. Participant Feedback

Participants responded extremely favourably towards Compass. 88.9% of participants reported that they found Compass easy to use and understand its responses. This feedback suggests that Compass allowed for the initiating and maintaining of natural conversation across this group of UX testers. In fact, participants were particularly impressed with the fact that they could have conversations with Compass “as if it were a human”. This aided in their ability to independently complete their work experience and skills obtaining journeys. 77.8% of participants found that Compass identified accurate and relevant skills without omitting any irrelevant skills. 88.9% of participants said that Compass helped them gain a better understanding of their experiences and skills, particularly because of how simply it put these in their summaries.

All participants welcomed the prospect of Compass suggesting jobs and sectors for them to apply to given their experiences. Additional feedback included having personal traits and characteristics as part of Compass skills summaries and including qualifications in the Compass CV. 100% of the participants who completed the UX test said they would recommend Compass to other job seekers. All of this taken together is strong evidence in favor of the hypothesis that participants are satisfied with their overall experience using Compass.

1. Positive Observations

Predictability. Over time, participants began to anticipate the flow of questions asked by Compass for each experience. This yielded quicker responses since participants could combine answers to two separate questions that they eventually knew would be asked by Compass (for example, what the experience was and where it was) without being prompted to do so.

Persistence. Compass’ persistence, particularly in the unseen economy, is a strength. Compass asks participants a series of headline questions to initially characterize their experience in the seen or unseen economy. After an experience is imputed by a participant, Compass repeats the question that has been answered before moving onto the next work experience category. This repetition of questions allows participants to note all of their work experience (and, in turn, their associated skills), within both the seen and unseen economy, which is particularly important for the unseen economy.

Agility. Throughout participants’ user journeys, Compass was able to effectively redirect the conversation back to topic when participants misunderstood questions. Similarly, Compass could navigate through most grammar and spelling mistakes, which each participant made at least one of. The only spelling mistake Compass did not automatically rectify was “Checkers”, which was erroneously spelt as “Checker” by Participant 4. This spelling error persisted until the experience summary was presented to the user. This notwithstanding, participants have several opportunities to correct spelling errors through their Compass user journeys.

Another instance of Compass’ agility is in the case of Participant 6, who incorrectly imputed a volunteer role as a paid role. When the participant responded to a question that asked if she had worked for a company or business for money, Compass identified key words, including “volunteer”, in her response and then proceeded to probe into whether her imputed experience was indeed paid or unpaid. After uncovering that it was an unpaid position, Compass was able to accurately classify this role as such and correctly list it Participant 6’s work experience summary.

1. Considerations

Misunderstandings. While Compass has yielded objectively great results across this cohort, it would be remiss to omit elements to consider when rolling it out to larger audiences. The first consideration is the language barrier effect. Compass requires a moderate proficiency in English. For the South African youth case, English is likely not the average prospective Compass users’ first language.

Each participant in the Tabiya-Harambee UX testing session possessed an average proficiency level high enough to get through the UX test and the questionnaire afterwards. However, there were times, such as in Participant 4’s UX test, when the moderator and participant conversed in the participant’s native language when the participant asked questions or when the moderator explained what Compass is. Moreover, it was clear that Compass posed some questions that caused some confusion, misunderstanding or hesitation among participants. Some examples include:

1. “Can you tell me about the first experience you had working for yourself?” To this question, Participant 1 described the independence and fulfilment he derived from this experience rather than describing the experience itself. Compass redirected the conversation by subsequently asking the participant what kind of work he participated in.

2. “...what was a typical day like at work” is the phrase used by Compass to initiate information on what skills participants obtained from their various work experiences. Participant 1 provided information related to the atmosphere or external happenings of a typical day rather than the tasks that they completed on a typical day. Compass redirected the conversation by asking what tasks participants were responsible for.

3. When Compass asked Participant 5 to “tell me what your experience was like”, the participant responded, “It was a good experience to volunteer”, rather than going into detail about what tasks the experience entailed.

Compass’ ability to infer findings and redirect conversations despite misinterpretations is a glaring strength in this regard. However, given that only 11% of participants completed Compass’ experiences and skills assignment sections in time, refining misunderstood questions to save time may be prudent. Compass’ ambition to expand its language offerings beyond English also bodes well for mitigating misunderstandings.

Confidentiality. A related but slightly different consideration worth mentioning is how Compass uses erroneously imputed, confidential information. Throughout each of the interviews conducted, Compass did not directly ask for sensitive or confidential information. However, there were times throughout the UX test where participants misinterpreted or erroneously answered one of Compass’ questions. In Participant 3’s case, his misunderstanding of the question “Can you tell me, was this a paid job?” caused him to impute the exact salary he earned from the job, rather than to affirm that the position was paid. This information did not appear on the Participant’s CV; therefore, it had no impact on his results for this version of Compass. However, if the ambition is to eventually roll out a version of Compass that uses the conversations it has with young people to suggest suitable jobs and sectors to find potential jobs, then this could be an issue. Compass could, in this case, provide a participant who has erroneously imputed their exact salary with job opportunities that pay within that same salary range and, in so doing, narrow their job application options relative to a participant whose remuneration is not known by Compass.

Skills Reporting. Participant 5 noted that “patience and accuracy” were skills required to do one of the roles he had well. Despite asking for this information, Compass omitted these skills from his skills summary. It is worth noting that Participant 5 felt that these skills were not as important as the skills surfaced by Compass. Conversely, Participant 7 noted that a great personality and good communication were essential for working at the company that he previously worked for. While neither of these were explicitly included in his skills summary, “customer service” - a plausible alternative to name these characteristics - was.

Whether or not to include skills explicitly mentioned by the participant is not easily answerable. For some participants, such as Participant 5, the skills summarized by Compass (which are based on the European Skills, Competences, Qualifications and Occupations (ESCO) framework) seem to resonate well with their experiences. In contrast, in other cases, such as the case of Participant 4, self-assigned personal characteristics and attributes would be a value addition in a CV. Nonetheless, if Compass explicitly asks participants to impute skills which they feel are important in their role, it seems worthwhile that Compass ought to, at the very least, find the most closely related ESCO skill to add to the participant’s skill summary.

In addition, when asked to provide information about her experience as a cashier in the work experience attribution section of Compass, Participant 4 noted that as a cashier, she was “assisting customers with electronic and cash handling payments, scanning and packing customers groceries, counting float and cash up”. Despite these detailed tasks and skills put forward by this participant, Compass did not include any of these in her work experiences summary. This is likely because she had not yet reached the portion of Compass that asks directly about tasks participants about their gained skills from a particular job. Unfortunately, Participant 4 did not proceed to the skills attribution section of Compass because of insufficient time to do so. This left Participant 4 feeling frustrated and disappointed. Ideally, Compass should be able to identify skills information given in the work experience section and preemptively add this skills information to a young person’s work experiences summary, so they do not have to repeat this information in the skills assignment section.

Experience Reporting. There are several instances where Compass erroneously summarizes participants' work experiences. For example, Participant 2’s unseen economy experience was duplicated in her work experience summary, and similarly, Participant 4’s cashier experience was erroneously duplicated.

One of Compass’ opening questions to Participant 6 was, “Have you ever worked for a company or someone else’s business for money?” Participant 6 responded to this question by saying, "Yes I am currently working for a company.” Compass then proceeded to gather information on the name and location of the company as well as how long she had worked there. However, Compass did not ask her what the title of her role was. Therefore, in her experiences summary, this work experience was named “Working for a Company”.

In another instance, Participant 5 erroneously classified waged employment as contract work (this is unsurprising since the term contract work could easily be interpreted as work done after signing a contract, especially if English is not a participant’s native language). During the dialogue, Compass rectified this error and correctly identified the participant’s experience as waged employment rather than contract work. However, Compass included “Contract work (Self-employment)” in the participant’s work experience summary despite this. Finally, Participant 5 worked remotely as a data capturer for a United Kingdom-based organization. When Compass summarized this experience, the fact that this experience was done remotely was omitted. Therefore, it may appear that the participant worked in the United Kingdom to a prospective employer.

Overlapping Experiences. Some South African youth hold more than one job at the same time to maximize their income. Often, one job may be held as a paid opportunity undertaken by an employer, while the other is a microentrepreneurship or freelance role that the young person independently does. Participant 1, for example, held a volunteer and a self-employment position while undertaking waged employment. Participant 2 mentioned that she would only ordinarily list some of the experience items she disclosed to Compass on her CV. It is useful to understand whether South African employers are more or less likely to hire someone with a CV that includes overlapping experiences relative to someone who performs one experience at a time. If an employer thought that overlapping self-employment (or volunteer) roles may create a conflict of interest or time mismanagement issue, even if this may not be the case, this could work against a jobseeker’s employment prospects. Alternatively, if an employer viewed holding multiple positions simultaneously as a signal of a hard-working nature, this could work in favor of a jobseeker. Whether a young person is allowed or encouraged to list overlapping full-time experiences ultimately ought to depend on the young person's objective and employees' expected responses.

Unseen Economy Bottlenecks. Compass is more susceptible to producing errors or yielding misunderstandings when it asks for unseen economy experiences relative to seen economy experiences. For example, Participant 2 was asked to give specific information about her experience with helping others; she said she was “well known in the community”. Compass, unsatisfied with this answer, then asked the question again in the same way. This led to the participant flagging this with the moderator. Ultimately, Compass erroneously duplicated the unseen economy experience provided by Participant 2 on her CV.

In another instance, when Participant 4 was asked whether she has “ever helped out friends or family members without getting paid”, she responded by describing an instance where she charitably gave her colleague money for transport and food. Compass failed to identify that this activity did not meet the requirements of an unseen economy activity and proceeded to gather more information to (erroneously) list this activity in the participant’s experience summary.

Participant 5’s unseen economy activity was taking his parents shopping. The Compass experience reporting structure urged the participant to provide a date when this activity happened. For the unseen economy, it can be particularly difficult to confine ongoing sporadic activities to a start date and end date. Indeed, it is plausible that Participant 5 took his parents shopping on several occasions despite reporting only the most recent date that he had done this activity. Moreover, Compass’ classification of the participant’s self-reported activity of taking is parents shopping as “helping out friends” in his work experience summary seems ill-defined.

The unseen economy is conceptually challenging to articulate non-technically. Moreover, assigning a “location, duration, organization name” structure to unseen economy may not work as seamlessly as in the seen economy. Hence, the current unseen economy prompts can do with revision in light of the above-mentioned bottlenecks.

External Validity. Finally, it is vital to understand the validity constraints of this exercise. These results hold for the participants who completed the UX test during the time they did so. However, due to the relatively small sample size of participants who completed the UX test, these results cannot be interpreted as applying to any other group.

Conclusion

Tabiya worked with Harambee Youth Employment Accelerator, to roll out a Compass UX Testing Session to nine South African youth who made up the session’s participants. The participants who completed the UX test had diverse demographics. Each session was managed and moderated by a Harambee staff member. In addition to using Compass on a smartphone, participants answered a series of questions to gauge how they felt about their interaction with Compass.

The feedback provided by participants is in overwhelming favor of the ease of use, accuracy and relevance of Compass in identifying their skills and experiences. The participants who used Compass were largely able to do so independently. This positive feedback is encouraging and extremely compelling, but it must be balanced with the considerations that must be made before rolling out Compass.

Compass is an innovative, AI-powered chatbot designed to revolutionize the way a young person identifies, articulates, and showcases their skills. Developed by Tabiya, Compass is a personal career assistant, helping youth uncover hidden talents and match them with the best opportunities in the job market.

The Compass Solution

Compass is an open-source, conversational AI tool that:

1. Engages in natural dialogue to explore your experience

2. Analyzes a user's input to identify and categorize skills against localized taxonomies

3. Identifies skills from both formal and informal work

4. Creates a comprehensive skills profile tailored to the user

5. Generates a professional customizable CV highlighting one’s strengths

6. Matches a user's skills with relevant economic opportunities

7. Recommends personalized recommendations for skill development and career advancement.

Compass uses a large language model (LLM) and a conversational interface to help job seekers build a CV that highlights their skills. The tool combines a commercial LLM with a human-reviewed skills taxonomy for the labor market.

Addressing Critical Challenges

In today's rapidly evolving job market, Compass tackles two persistent and interconnected problems that hinder effective workforce development:

• Challenges in showcasing skills: Crafting a CV that highlights relevant skills is tough for many job seekers. It's about translating experiences into terms that appeal to employers. This is harder for those with non-traditional or informal experience. A poor CV can lead to missed opportunities.

• The scalability struggle: Traditional methods rely heavily on human career counselors. While these professionals offer valuable insights, this approach has significant limitations.

Compass breaks down barriers by offering an AI solution that's scalable, affordable, and high-quality.

Benefits of Compass

Upcoming Features

• Expanded geographic reach: Widespread adoption of Compass for use in emerging markets

• Multilingual support: Expand language capabilities to serve diverse populations

• Voice integration: Enable our users to speak to Compass in their local dialects

• Portable skills wallet: Compass skills exploration outputs activated in an interoperable skills wallet

• Enhanced features: Develop more advanced career pathing and skills development recommendations

• Integration capabilities: Create APIs and tools for seamless integration with existing job platforms and career services

• Impact measurement: Implement a multi-site randomized control trial to rigorously estimate impacts

Our Open Source Commitment

Compass is designed as a digital public good:

• The core technology is open-source, allowing for transparency and community-driven improvements

• We aim to build a diverse global community of contributors and implementers around Compass

• Organizations supporting youth in their career journeys can adapt Compass to their specific needs and contexts

Get Involved

We welcome partnerships and collaborations to further develop and implement Compass:

• For inquiries about supporting or implementing Compass, connect with us here

• To learn more about our current projects and future plans, visit https://www.tabiya.org

• Follow our progress and join the conversation on https://www.linkedin.com/company/tabiya

Together, we can leverage the power of AI to create more inclusive and efficient labor markets worldwide.

Funders and Partners

Our vision for Compass includes:

• Expanded geographic reach: Widespread adoption of Compass for use in emerging markets

• Multilingual support: Expand language capabilities to serve diverse populations

• Voice integration: Enable our users to speak to Compass in their local dialects

• Portable skills wallet: Compass skills exploration outputs activated in an interoperable skills wallet

• Enhanced features: Develop more advanced career pathing and skills development recommendations

• Integration capabilities: Create APIs and tools for seamless integration with existing job platforms and career services

• Impact measurement: Implement a multi-side randomised control trial to estimate Compass impact on career outcomes

Introduction and Warm-up

Hi [participant’s name], thank you so much for taking the time to join me today!

My name is [moderator name], and I'm helping with a research project to understand how a new tool called Compass might be able to assist job seekers in their job search journey.

Compass uses research and AI to help job seekers explore their skills and provides guidance on how best to leverage them in their job search.

Your feedback will help us make improvements to Compass so it can work better for job seekers like yourself.

Participant's consent: did you get a chance to sign the consent form?

• Everything you share in this session will be kept confidential. We won't use your name or any identifying details in our reports.

• So please feel comfortable sharing openly and honestly

• Also, there are no right or wrong answers, the most helpful to us is to hear your perspective

• Please also keep everything you see and everything we talk about today confidential

Permission to record: I have a small request; would it be okay with you if I record our session?

• I’ll be engaged in the conversation with you and want to make sure I don’t miss anything important that you tell me, only myself and the team will be able to see it and we delete everything after the completion of the study

Permission to livestream: I have one more request; would it be okay with you if I livestream our session?

• Members of the team working on Compass are eager to learn from your experience and they would love to observe the session.

Thank you so much, much appreciated [if they say yes]

No worries at all [if they say no]

Do you have any questions for me?

Before we jump into trying out Compass, I'd love to hear a bit about your experience looking for a job.

• Can you briefly tell me a little about your job search so far?

Thank you for sharing a bit about your job search journey, let’s get started with Compass.

Tasks

Task 1: Initial Interaction [10 min]

• Scenario [Moderator to read this part]:

• You are a job seeker exploring finding a job. You want to find work that suits you, based on your skillset. Start a conversation with Compass to identify your skills.

• Instructions: [Moderator to read this part] (to avoid bias, alternate between the two options as you go through the interviews)

• Option 1: how would you go about using Compass to identify your skills.

• Option 2: introduce yourself and tell Compass you are looking to identify your skills for your job search.

• Feedback: [Moderator to read this part]:

• What do you think of the conversation flow so far?

• On a scale of 1 to 5, how satisfied are you with this conversation with Compass so far? (1 unhelpful 5 very helpful)

• Success criteria: participant successfully initiates a conversation with Compass.

Task 2: Skill Identification [10 min]

• Scenario [Moderator to read this part]:

• As you can see, Compass has asked you questions about your various experiences

• Instructions: [Moderator to read this part]:

• Go ahead and answer Compass's questions about your past roles and responsibilities to the best of your ability

• Feedback: [Moderator to read this part]:

• What do you think of how Compass is trying to identify your skills?

• On a scale of 1 to 5, how satisfied are you with this conversation with Compass so far? (1 unhelpful 5 very helpful)

• Success criteria: participant answers questions about their work experience and engages in a dialogue with Compass

Task 3: Skill Review & Feedback [10 min]

• Scenario [Moderator to read this part]:

• As you can see, Compass has generated a list of your potential skills

• Instructions: [Moderator to read this part]:

• Go ahead and take a moment to review these skills identified by Compass

• Feedback: [Moderator to read this part]:

• How accurate/relevant do you feel the skills Compass identified for you are? Why or why not?

• On a scale of 1 to 5, how helpful was Compass in identifying your skills? (1 unhelpful, 5 very helpful)

• On a scale of 1 to 5, how satisfied would you say you are with Compass’ ability to identify your skills?

• Success criteria: participant reviews the generated skills, provides feedback, and clarifies any discrepancies

Post-Test Feedback Questions [10 min]

• [Ease of use] How easy was it to interact with Compass and understand its responses?

• [Accuracy & relevance] How accurate and relevant do you feel the identified skills are? Why?

• [Accuracy & relevance] Are there any skills that Compass incorrectly identified for you?

• [Accuracy & relevance] Are there any skills you have that Compass did not identify?

• Did Compass help you gain a clearer understanding of your skills? why or why not?

• Would you recommend Compass to other job seekers? Why or why not?

• In the future, Compass will provide you with advice on which sectors you could focus on for your job searches based on the skills it will have identified for you. What do you think about this?

• In the future, Compass will match you with jobs that align with your identified skills. What do you think about this?

• Any other feedback or suggestions for improving Compass?

Thank you and close