Experience has shown that young jobseekers struggle to identify skills that are the most relevant to their professional occupations and hobbies, or the skills at which they are the most proficient. Yet, helping them navigate ill-functioning labour markets entails surfacing the full length of their human capital – namely, the full set of skills they possess -, while helping them to present it in a strategic manner to improve their job seeking perspectives. To help jobseekers do so, Harambee hopes to rely on static surveys prompting young jobseekers to select, for each of their occupation / activity, the five skills that they are the most proficient at. To do so, they will be presented with lists of ESCO skills associated with their activities, or with skills that have been selected through the work of Harambee and the Oxford Martin School (for micro-entrepreneurship and unseen activities). This raises the question of the optimal way to present skills to job seekers, to ensure that 1 – we cover their whole human capital, 2 – they can effectively pick relevant skills to populate their inclusive CVs, 3 – this process is efficient. Yet, evidence has shown that the way individuals are prompted to pick answers out of list of options impact their final decisions. In this note, I summarize some research pieces to inform how the lists of skills submitted to jobseekers by Harambee and OMS may influence the decisions.

Choice Overload and Underload

These notes are based on « Cognitive and Affective Consequences of Information and Choice Overload” by Reutskaja et al.

Harambee has expressed concerns that the number of options presented to job seekers may overwhelm them, making their experience on the platform anxiety inducing and the results inefficient. These concerns are aligned with insights from behavioural science, focusing on the concept of “choice overload”. Intuitively, individuals presented with too many options to pick from may find it harder to pick ad experience anxiety, unsatisfaction, loss of motivation to pick, stress, or even choice paralysis. It is to be noted nevertheless that the reverse phenomenon, choice underload, may lead to frustration as well, as the desired options may not be presented to individuals. Behavioural theory suggests that the level of satisfaction felt by individuals prompter to pick a choice out of several options follows an inverted U-shape. Namely, the optimal number of options to give individuals is not the maximum.

The intuition behind this is the following. When presented with an increasing number of options, the needs of people are more likely to be satisfied. In our work, more options means that Harambee’s young job seekers are more likely to find the skills that they are the most of proficient at among the ones we present them with. This abundance of choices can also be beneficial from a psychological point of view. In our case, it may comfort users into the idea that they have large sets of skills, and thus allow them to have a more positive approach in their job search. On the other hand, large sets of options have psychological and time costs. Studies have shown that our minds do not account for the full sets of available information when taking decisions, but only subsets (Wedel and Pieter, 2007). Following Reutskaja et al’s terminology, this “inattentional blindness” means that jobseekers presented with long lists of skills risk ignoring skills that they would have picked in smaller sets of options. In the case of Harambee users, a large set of options may also induce stress, especially linked to the sentiment of forgoing important skills. Worse, it may lead to a loss of motivation to answer the prompt correctly, and to users selecting skills by default. This evidence suggest that Harambee is right to worry about the number of skills presented to young jobseekers being too large.

Ordering of Skills

Once the lists of skills presented to young jobseekers have been narrowed down, the order in which the skills are presented is likely to impact their picks. As Harambee and the teams in Oxford are considering ordering the skills based on concepts such as transferability (essentially, the frequency with which a skill is associated with an occupation in ESCO), understanding how that order may influence users’ pick is critical. Intuitively, one could think that users tend to pick the first skill presented to them, especially when they must take an action (click “see more”, for instance) to see other options. Studies researching the effect of the order of questions in online surveys suggest that order does matter. A study conducted in the US in 2008 found that low-education respondents who answer surveys quickly – which is likely to be the case of Harambee users – a most prone to primacy effects (bias toward selecting earlier response choices). A study conducted in Xiao et al in 2024 also observes such bias, while finding that it is inexistent in “objective questions”, for instance those pertaining to demographics.

The implications for the Harambee platform are not straightforward. On the one hand, Harambee may seek to nudge job seekers into picking skills that Harambee’s counsellors consider as better picks. This would imply putting these skills on top of the list of options. On the other hand, Harambee may choose not to try and influence users’ picks. However, a random order would effectively lead to answers biased toward the first options presented. Countering this bias is not evident, which may be another argument to choose to hierarchize skills.

Prompt

The evidence previously quoted finally suggest that choice overload is different from information overload. Namely, the sentiment of discomfort felt by individuals in choice overload does not come from the number of options only, but also from being prompted to choose a subset of options. Intuitively, the disproportion of the option set with the allowed number of picks – too many options, too few picks – makes choosing uncomfortable. This is likely to be especially true in the case of Harambee users selecting skills, as choosing skills to populate a CV is consequential. Being prompted to choose 5 skills is likely to create two issues: 1 – Encourage people to select 5 skills although they are not proficient in some of them. Picking less skills may make users lose confidence by suggesting they have few skills. 2 – Make users feel like they are not able to surface enough of their human capital, which may create frustration.

However, it is unrealistic to populate CV’s with more than 5 skills per occupation. Moreover, the occupation title also sends signals to employers about jobseekers’ skill sets. Therefore, we only recommend adjusting the prompt from “Pick the five skills that you are the most proficient at” to “Pick up to five skills that you are the most proficient at”.

All in all, evidence appear to justify narrowing down the lists of options presented to young jobseekers on the Harambee platform. It also gives evidence that the order of options matters, which may inform decisions taken by Harambee. Finally, it suggests choosing a prompt carefully, so as to not induce stress or demotivation on the platform’s users.

Bibliography

Neil Malhotra, Completion Time and Response Order Effects in Web Surveys, Public Opinion Quarterly, Volume 72, Issue 5, December 2008, Pages 914–934, https://doi.org/10.1093/poq/nfn050.

Reutskaja, E., Iyengar, S., Fasolo, B., & Misuraca, R. (2020). Cognitive and affective consequences of information and choice overload. In Routledge handbook of bounded rationality (pp. 625-636). Routledge.

Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.

Wedel, M., & Pieters, R. (Eds.) (2007). Visual Marketing: From attention to action. Hove: Psychology Press.

Xiao, Yaqing and Yan, Hongjun and Davidson, Erik, Response Order Biases in Economic Surveys (February 2, 2024). Available at SSRN: https://ssrn.com/abstract=3786894 or http://dx.doi.org/10.2139/ssrn.3786894.

This page summarises a proposal for how to implement A/B testing of how to elicit skills for SAyouth.mobi users’ inclusive CV. This research is motivated by concerns that users may struggle to select between different skills, and an interest in comparing different potential metrics for skill priority. The proposed research is an on-platform randomised evaluation of how users are asked to capture the skills relating to their work experiences.

Background

In the search to highlight and surface the skills South African youth have as a result of the activities they have performed and are currently involved in, we conducted a survey that asked microentrepreneurs to rate the importance of different skills associated with their occupation. Specifically, we asked about 1500 microentrepreneurs “On the 1 to 5 scale where 1 is ‘Not important at all’ and 5 is ‘Very important’, how important is [skill tag] to your work?”. The results showed a tendency for participants to assign high importance to all skills presented to them.

This poses a potential challenge for the Harambee platform. If job seekers prioritize all skills equally, they might be overly reliant on the initial set of skills they encounter during their platform search. This could lead them to select skills that aren't necessarily the most relevant for their desired careers, hindering their job search effectiveness. To address this concern, our research aims to investigate how the order of skill presentation on the Harambee platform influences job seeker choices.

Thus we seek to answer the following questions:

1. What is the impact of presenting skills to platform users in different orders on the skills selected?

2. Is there an order that offers users the best support in their job search journey?

3. Do different skill priorities have downstream impacts on job search patterns?

4. Do different skill priorities have an impact on employer interest in the candidate?

5. How often should we ask users to update their skills?

Proposal

We propose implementing on-platform A/B testing of how to surface skills to users. This testing should help us to evaluate how users respond to different presentations of skills and consequently refine how best to elicit skill reporting from platform users.

The ideal method for testing skills surfacing is evaluating actual user skill selections on platform. This would be achieved by comparing how similar (statistically comparable) users select skills when presented with different list orders and lists of varying numbers of skills. Thus, we wish to randomly assign users to be presented with different lists when they are asked to select the skills they wish to include on their CV.

We propose a number of potential skill priority metrics:

1. Random This serves as a neutral benchmark. It is also the most theoretically “hands off”.

2. Transferability score Transferability score = number of appearances of this skill in ESCO / number of occupations in ESCO . This may nudge users towards listing skills that are commonly used in many other opportunities, or in jobs commonly listed on the platform. If users are encouraged to select skills that are more transferable, we may be able to improve the visibility of skills that firms are interested in knowing about.

3. Inverse scarcity of skills Inverse scarcity of skills = inverse of number of other users with that score, conditional on the number being above 10 (to rule out idiosyncratic skills). More rare skills may also help job seekers to stand out more from the rest of the field of applicants, and help firms differentiate between job seekers more easily.

4. Some weighted average of these goals Its likely some mixture of these priorities is appropriate. It may also be that different jobs or different fields may require different weightings of the set of priorities. Developing a preferred weighted average may require some desk work to get input from field experts and literature about relative importance of different measures for particular industries or worker experience profiles.

5. “Other” field to search for skills Offering users the chance to describe their own skills offers another extreme hands off benchmark. This arm can also be used to validate the accessibility of the language used by the taxonomy to describe the skills surfaced to users.

6. More skill options For each of the above options we propose surfacing five skills. In this option we propose extending the list to surface more skill options for users to choose from.

We will follow a stratified random sampling strategy based on gender and the type of work the individual has performed (i.e. currently unemployed, microentrepreneur, and employed). The sample will be randomly assigned into treatment groups and each group will be presented with a pre-ranked set of skills. The testing could run for one to three weeks depending on logistical constraints for Harambee and the target sample size for each treatment cell. A/B testing in principle does not require detailed power calculations because patterns are informative even if they do not cross the threshold for academic statistically certainty. Samples as small as 20 users per arm could already offer indicative patterns, although on-platform may make it easy to reach far more users. A larger sample would naturally offer the advantages of more precise estimates of effect and the opportunity to account for the role demographic characteristics may play in mediating responses to different skills list orderings. We propose two levels of randomisation.

Across Users Effect

Randomise at the day/week level. While we prefer a day level randomisation, we are happy to work with logistical constraints of bringing various lists live on the platform. We could do every two days, or every week if one of these is easier. Each new randomisation block we will apply a different skill priority ranking. Cycle through each of the treatment arms repeatedly over 3 weeks. This will generate a treatment arm for each list priority order and therefore enable the central comparison of different potential skill priorities.

Within Users Effect

Once a month prompt an opportunity to review skills reported previously and surface skills in a randomly chosen priority (may be the same or different priority users have previously been asked). This will enable us to examine whether the same user reacts differently to receiving skills in different orders, and for users who are shown the same list twice, how stable elicitation is. A measure of skill reporting stability is useful because it can be used to decide how often the platform should prompt users to update their skills or CV. There are a number of reasons to believe that this will be less useful information than the primary randomisation, however, it will be important that the platform has the technical capacity to encourage users to modify which skills they are capturing in their inclusive CV so that we can implement the findings of the A/B testing for platform users in the control group.

Measurement

Many of the indicators of which list to prefer can be collected from administrative data already on the platform. The primary measure of impact is the simple comparison of skills chosen under different treatment conditions. Statistical properties of the skills chosen offer information a number of dimensions of choice quality:

1. Correlation between skills chosen under different list orders can tell us the degree to which the different orders make a difference to the skills ultimately chosen.

2. Variation in skills chosen within a single priority type can tell us whether all individuals are behaving the same way, indicative of mechanical clicking/less engagement with individual matches with skills or thoughtful engagement.

3. Measures of how often the first three skills, or the first skill, or the last skill, or the skill most centrally displayed on the screen can capture indicators or mechanical clicking, or reduced engagement.

4. We could manually review a subset of respondents to evaluate skill choice against their profile.

5. UX measures of time to complete fields, number who start and abandon part way through.

Demographics which may mediate how users interact with skill lists can also be drawn from user profiles. We would likely focus on:

• Gender

• Date of birth (Month and Year)

• Occupation/activity

• Education level

• Geographic location

• Experience history (previous years employed)

• Disability status

All of this information is available for users on the platform, so it would simply be a question of pulling the relevant administrative data.

If there is interest in using downstream outcomes to adjudicate between list priorities, we could also draw data on which jobs individuals who have been exposed to different lists click on and apply for. We could also ask firms to select skills they would like to see prioritized among platform applicants from similarly ordered lists to obtain a measure of firm preferences.

We could also collect additional data either using on-platform prompts, off-platform follow up surveys, or some small focus groups. Additional data would enable us to measure user satisfaction with the different lists, elicit user criteria for how they select between skills and evaluate how confident or committed users are about the skills they selected.

The optimal way to use our inclusive taxonomy to help young jobseekers is not obvious. For instance, it could be used as the basis of a static survey in which users pick occupations and skills manually, or in a more flexible way by using AI. Although certainly complementary, we hope to test these two approaches separately to identify the advantages and disadvantages of each approach.