UX Evaluation

In August 2024, Tabiya tested the user experience of Compass with job-seekers recruited from Harambee Youth Employment Accelerator.

Background and Motivation

Tabiya, in partnership with Harambee Youth Employment Accelerator, conducted a three-day series of user experience (UX) tests devised to answer the following questions:

1. Can participants initiate and maintain a natural conversation with Compass?

2. Does Compass effectively guide users through the skill identification process?

3. Do users find the skills identified by Compass relevant and accurate?

4. Are participants satisfied with the overall experience and feel they have a clearer understanding of their skills?

. This is due, in part, to. That is, work-seekers may have incomplete information about their skills. This means that work-seekers could apply for jobs that they are incompatible with or . Compass is built to mitigate this particular search distortion, which forms part of the drivers of unemployment, especially amongst the youth.

Methodology

South African youth (between the ages of 18 and 35) subscribed to Harambee’s SA Youth Mobi Platform were shown an advertisement to join a UX testing session hosted at Harambee offices in Cape Town. Upon receipt, 604 were deemed most compatible with the selection criteria. This criterion includes the following:

1. Young person is subscribed to SA Youth Mobi Platform

2. Young person ordinarily resides no more than roughly 15 kilometeres away from the Harambee Cape Town office

3. Young person has had work experience before

4. Young person is actively looking for employment

From this criterion, black female participants and participants who had evidence of starting their own small business were favored in the final UX test participant selection.

Before the UX test session begins, each participant is given a consent form to read and sign. Where applicable, participants were asked to sign a photography consent form if they were comfortable doing so. Following this, each UX test comprised four parts. The first part consisted of an introduction and asking the participant for permission to record the session. This part of the UX testing session was also used to explain what Compass is and how it works to the participant.

The second part consisted of the commencement of the UX test. During this part, each participant interacted with Compass as independently as possible until their experiences and skills were summarized, or until the moderator indicated they should stop interacting with Compass due to a time constraint. The UX test involved Compass gathering information from participants on their work experiences, including paid and unpaid work in establishments or non-establishment settings. This is referred to as the work experience section throughout this report. Following this, Compass produced an experience summary listing the participants' experiences, locations, and durations chronologically. Next, Compass gathered information on what skills participants had obtained from each work experience through an interactive dialogue. Up to five skills per experience were subsequently listed at the end of the dialogue. This is referred to as the skills assignment section throughout this report.

During the third part of the session, the moderator asked the participant a series of post-test feedback questions. Finally, the moderator closed the session by giving each participant the timeframe to expect their monetary incentive (of R450) for participating in the UX test. The closing was also used to ask participants if they were comfortable with being placed in a Harambee WhatsApp group for an expert panel that is periodically invited to participate in Harambee sessions targeted at its youth database. All four parts were projected to take 45 minutes but could take up to one hour. In addition to R450, each participant was given a snack pack before each UX test. This is done to ensure that participants had food to eat before commencing their UX tests. The complete discussion guide for each UX test can be found here.

Compass was designed to extract information from its users conversationally. As such, Compass functions similarly to a messaging application. Compass textually asks its users for information pertaining to their experiences and allows users to respond before concluding or probing for more information. Given that this exercise operates similarly to an interaction where someone would communicate through text messaging on their smartphone, the UX test was conducted on a cellphone. For heightened accessibility and ease of use, the cellphone chosen was the Samsung Galaxy A23 because this is the most widely used smartphone among the youth in Harambee’s database.

Results

1. Demographics

A set of 11 SA Youth subscribed youth were selected to test the functionality of Compass. Three of these participants tested out Compass on the 27th of August. Two were male, and one was female. Four were scheduled to complete the testing on the 28th, but there was an attrition rate of 50% on this day. A male and a female showed up; the two not in attendance were male. The 29th of August was the final day of testing. Four participants, three female and one male, each tested Compass. Therefore, 82% (9 people) of the chosen cohort of 11 people participated in the UX Testing. 44% of the UX testers participants were male and 66% female.

1. Timing

From the nine young people who tested Compass, two groups emerged. Group 1 comprised three participants who received a work experience summary but not a skills summary. Group 2 included six participants who received both work experience and skills summary. Group 1 took an average of 32 minutes to complete the entire UX testing session. Group 2 took an average of 32 minutes and 3o seconds to complete. Overviews of each UX testing session, including details on the timings of each session can be found here.

The full Compass experience was completed without interruption only once. This was done by Participant 9 who only possessed one experience and thus, one set of skills. Hence, the initially projected time of 45 minutes to complete the entire UX testing session was an underestimation. Instead, all participants finished within 45 minutes only because the moderator of the participant sessions prematurely instructed participants to finish their interactions with Compass. Given that participants were paid for their time, not doing this would have violated labor laws.

1. Participant Feedback

Participants responded extremely favourably towards Compass. 88.9% of participants reported that they found Compass easy to use and understand its responses. This feedback suggests that Compass allowed for the initiating and maintaining of natural conversation across this group of UX testers. In fact, participants were particularly impressed with the fact that they could have conversations with Compass “as if it were a human”. This aided in their ability to independently complete their work experience and skills obtaining journeys. 77.8% of participants found that Compass identified accurate and relevant skills without omitting any irrelevant skills. 88.9% of participants said that Compass helped them gain a better understanding of their experiences and skills, particularly because of how simply it put these in their summaries.

All participants welcomed the prospect of Compass suggesting jobs and sectors for them to apply to given their experiences. Additional feedback included having personal traits and characteristics as part of Compass skills summaries and including qualifications in the Compass CV. 100% of the participants who completed the UX test said they would recommend Compass to other job seekers. All of this taken together is strong evidence in favor of the hypothesis that participants are satisfied with their overall experience using Compass.

1. Positive Observations

Predictability. Over time, participants began to anticipate the flow of questions asked by Compass for each experience. This yielded quicker responses since participants could combine answers to two separate questions that they eventually knew would be asked by Compass (for example, what the experience was and where it was) without being prompted to do so.

Persistence. Compass’ persistence, particularly in the unseen economy, is a strength. Compass asks participants a series of headline questions to initially characterize their experience in the seen or unseen economy. After an experience is imputed by a participant, Compass repeats the question that has been answered before moving onto the next work experience category. This repetition of questions allows participants to note all of their work experience (and, in turn, their associated skills), within both the seen and unseen economy, which is particularly important for the unseen economy.

Agility. Throughout participants’ user journeys, Compass was able to effectively redirect the conversation back to topic when participants misunderstood questions. Similarly, Compass could navigate through most grammar and spelling mistakes, which each participant made at least one of. The only spelling mistake Compass did not automatically rectify was “Checkers”, which was erroneously spelt as “Checker” by Participant 4. This spelling error persisted until the experience summary was presented to the user. This notwithstanding, participants have several opportunities to correct spelling errors through their Compass user journeys.

Another instance of Compass’ agility is in the case of Participant 6, who incorrectly imputed a volunteer role as a paid role. When the participant responded to a question that asked if she had worked for a company or business for money, Compass identified key words, including “volunteer”, in her response and then proceeded to probe into whether her imputed experience was indeed paid or unpaid. After uncovering that it was an unpaid position, Compass was able to accurately classify this role as such and correctly list it Participant 6’s work experience summary.

1. Considerations

Misunderstandings. While Compass has yielded objectively great results across this cohort, it would be remiss to omit elements to consider when rolling it out to larger audiences. The first consideration is the language barrier effect. Compass requires a moderate proficiency in English. For the South African youth case, English is likely not the average prospective Compass users’ first language.

Each participant in the Tabiya-Harambee UX testing session possessed an average proficiency level high enough to get through the UX test and the questionnaire afterwards. However, there were times, such as in Participant 4’s UX test, when the moderator and participant conversed in the participant’s native language when the participant asked questions or when the moderator explained what Compass is. Moreover, it was clear that Compass posed some questions that caused some confusion, misunderstanding or hesitation among participants. Some examples include:

1. “Can you tell me about the first experience you had working for yourself?” To this question, Participant 1 described the independence and fulfilment he derived from this experience rather than describing the experience itself. Compass redirected the conversation by subsequently asking the participant what kind of work he participated in.

2. “...what was a typical day like at work” is the phrase used by Compass to initiate information on what skills participants obtained from their various work experiences. Participant 1 provided information related to the atmosphere or external happenings of a typical day rather than the tasks that they completed on a typical day. Compass redirected the conversation by asking what tasks participants were responsible for.

3. When Compass asked Participant 5 to “tell me what your experience was like”, the participant responded, “It was a good experience to volunteer”, rather than going into detail about what tasks the experience entailed.

Compass’ ability to infer findings and redirect conversations despite misinterpretations is a glaring strength in this regard. However, given that only 11% of participants completed Compass’ experiences and skills assignment sections in time, refining misunderstood questions to save time may be prudent. Compass’ ambition to expand its language offerings beyond English also bodes well for mitigating misunderstandings.

Confidentiality. A related but slightly different consideration worth mentioning is how Compass uses erroneously imputed, confidential information. Throughout each of the interviews conducted, Compass did not directly ask for sensitive or confidential information. However, there were times throughout the UX test where participants misinterpreted or erroneously answered one of Compass’ questions. In Participant 3’s case, his misunderstanding of the question “Can you tell me, was this a paid job?” caused him to impute the exact salary he earned from the job, rather than to affirm that the position was paid. This information did not appear on the Participant’s CV; therefore, it had no impact on his results for this version of Compass. However, if the ambition is to eventually roll out a version of Compass that uses the conversations it has with young people to suggest suitable jobs and sectors to find potential jobs, then this could be an issue. Compass could, in this case, provide a participant who has erroneously imputed their exact salary with job opportunities that pay within that same salary range and, in so doing, narrow their job application options relative to a participant whose remuneration is not known by Compass.

Skills Reporting. Participant 5 noted that “patience and accuracy” were skills required to do one of the roles he had well. Despite asking for this information, Compass omitted these skills from his skills summary. It is worth noting that Participant 5 felt that these skills were not as important as the skills surfaced by Compass. Conversely, Participant 7 noted that a great personality and good communication were essential for working at the company that he previously worked for. While neither of these were explicitly included in his skills summary, “customer service” - a plausible alternative to name these characteristics - was.

Whether or not to include skills explicitly mentioned by the participant is not easily answerable. For some participants, such as Participant 5, the skills summarized by Compass (which are based on the European Skills, Competences, Qualifications and Occupations (ESCO) framework) seem to resonate well with their experiences. In contrast, in other cases, such as the case of Participant 4, self-assigned personal characteristics and attributes would be a value addition in a CV. Nonetheless, if Compass explicitly asks participants to impute skills which they feel are important in their role, it seems worthwhile that Compass ought to, at the very least, find the most closely related ESCO skill to add to the participant’s skill summary.

In addition, when asked to provide information about her experience as a cashier in the work experience attribution section of Compass, Participant 4 noted that as a cashier, she was “assisting customers with electronic and cash handling payments, scanning and packing customers groceries, counting float and cash up”. Despite these detailed tasks and skills put forward by this participant, Compass did not include any of these in her work experiences summary. This is likely because she had not yet reached the portion of Compass that asks directly about tasks participants about their gained skills from a particular job. Unfortunately, Participant 4 did not proceed to the skills attribution section of Compass because of insufficient time to do so. This left Participant 4 feeling frustrated and disappointed. Ideally, Compass should be able to identify skills information given in the work experience section and preemptively add this skills information to a young person’s work experiences summary, so they do not have to repeat this information in the skills assignment section.

Experience Reporting. There are several instances where Compass erroneously summarizes participants' work experiences. For example, Participant 2’s unseen economy experience was duplicated in her work experience summary, and similarly, Participant 4’s cashier experience was erroneously duplicated.

One of Compass’ opening questions to Participant 6 was, “Have you ever worked for a company or someone else’s business for money?” Participant 6 responded to this question by saying, "Yes I am currently working for a company.” Compass then proceeded to gather information on the name and location of the company as well as how long she had worked there. However, Compass did not ask her what the title of her role was. Therefore, in her experiences summary, this work experience was named “Working for a Company”.

In another instance, Participant 5 erroneously classified waged employment as contract work (this is unsurprising since the term contract work could easily be interpreted as work done after signing a contract, especially if English is not a participant’s native language). During the dialogue, Compass rectified this error and correctly identified the participant’s experience as waged employment rather than contract work. However, Compass included “Contract work (Self-employment)” in the participant’s work experience summary despite this. Finally, Participant 5 worked remotely as a data capturer for a United Kingdom-based organization. When Compass summarized this experience, the fact that this experience was done remotely was omitted. Therefore, it may appear that the participant worked in the United Kingdom to a prospective employer.

Overlapping Experiences. Some South African youth hold more than one job at the same time to maximize their income. Often, one job may be held as a paid opportunity undertaken by an employer, while the other is a microentrepreneurship or freelance role that the young person independently does. Participant 1, for example, held a volunteer and a self-employment position while undertaking waged employment. Participant 2 mentioned that she would only ordinarily list some of the experience items she disclosed to Compass on her CV. It is useful to understand whether South African employers are more or less likely to hire someone with a CV that includes overlapping experiences relative to someone who performs one experience at a time. If an employer thought that overlapping self-employment (or volunteer) roles may create a conflict of interest or time mismanagement issue, even if this may not be the case, this could work against a jobseeker’s employment prospects. Alternatively, if an employer viewed holding multiple positions simultaneously as a signal of a hard-working nature, this could work in favor of a jobseeker. Whether a young person is allowed or encouraged to list overlapping full-time experiences ultimately ought to depend on the young person's objective and employees' expected responses.

Unseen Economy Bottlenecks. Compass is more susceptible to producing errors or yielding misunderstandings when it asks for unseen economy experiences relative to seen economy experiences. For example, Participant 2 was asked to give specific information about her experience with helping others; she said she was “well known in the community”. Compass, unsatisfied with this answer, then asked the question again in the same way. This led to the participant flagging this with the moderator. Ultimately, Compass erroneously duplicated the unseen economy experience provided by Participant 2 on her CV.

In another instance, when Participant 4 was asked whether she has “ever helped out friends or family members without getting paid”, she responded by describing an instance where she charitably gave her colleague money for transport and food. Compass failed to identify that this activity did not meet the requirements of an unseen economy activity and proceeded to gather more information to (erroneously) list this activity in the participant’s experience summary.

Participant 5’s unseen economy activity was taking his parents shopping. The Compass experience reporting structure urged the participant to provide a date when this activity happened. For the unseen economy, it can be particularly difficult to confine ongoing sporadic activities to a start date and end date. Indeed, it is plausible that Participant 5 took his parents shopping on several occasions despite reporting only the most recent date that he had done this activity. Moreover, Compass’ classification of the participant’s self-reported activity of taking is parents shopping as “helping out friends” in his work experience summary seems ill-defined.

The unseen economy is conceptually challenging to articulate non-technically. Moreover, assigning a “location, duration, organization name” structure to unseen economy may not work as seamlessly as in the seen economy. Hence, the current unseen economy prompts can do with revision in light of the above-mentioned bottlenecks.

External Validity. Finally, it is vital to understand the validity constraints of this exercise. These results hold for the participants who completed the UX test during the time they did so. However, due to the relatively small sample size of participants who completed the UX test, these results cannot be interpreted as applying to any other group.

Conclusion

Tabiya worked with Harambee Youth Employment Accelerator, to roll out a Compass UX Testing Session to nine South African youth who made up the session’s participants. The participants who completed the UX test had diverse demographics. Each session was managed and moderated by a Harambee staff member. In addition to using Compass on a smartphone, participants answered a series of questions to gauge how they felt about their interaction with Compass.

The feedback provided by participants is in overwhelming favor of the ease of use, accuracy and relevance of Compass in identifying their skills and experiences. The participants who used Compass were largely able to do so independently. This positive feedback is encouraging and extremely compelling, but it must be balanced with the considerations that must be made before rolling out Compass.