Ordering of Skills: Review of Evidence

There is ample evidence that the way options are presented to users influence their picks. Understand users' biases is critical to improve the functioning of the Harambee platform.

Experience has shown that young jobseekers struggle to identify skills that are the most relevant to their professional occupations and hobbies, or the skills at which they are the most proficient. Yet, helping them navigate ill-functioning labour markets entails surfacing the full length of their human capital – namely, the full set of skills they possess -, while helping them to present it in a strategic manner to improve their job seeking perspectives. To help jobseekers do so, Harambee hopes to rely on static surveys prompting young jobseekers to select, for each of their occupation / activity, the five skills that they are the most proficient at. To do so, they will be presented with lists of ESCO skills associated with their activities, or with skills that have been selected through the work of Harambee and the Oxford Martin School (for micro-entrepreneurship and unseen activities). This raises the question of the optimal way to present skills to job seekers, to ensure that 1 – we cover their whole human capital, 2 – they can effectively pick relevant skills to populate their inclusive CVs, 3 – this process is efficient. Yet, evidence has shown that the way individuals are prompted to pick answers out of list of options impact their final decisions. In this note, I summarize some research pieces to inform how the lists of skills submitted to jobseekers by Harambee and OMS may influence the decisions.

Choice Overload and Underload

These notes are based on « Cognitive and Affective Consequences of Information and Choice Overload” by Reutskaja et al.

Harambee has expressed concerns that the number of options presented to job seekers may overwhelm them, making their experience on the platform anxiety inducing and the results inefficient. These concerns are aligned with insights from behavioural science, focusing on the concept of “choice overload”. Intuitively, individuals presented with too many options to pick from may find it harder to pick ad experience anxiety, unsatisfaction, loss of motivation to pick, stress, or even choice paralysis. It is to be noted nevertheless that the reverse phenomenon, choice underload, may lead to frustration as well, as the desired options may not be presented to individuals. Behavioural theory suggests that the level of satisfaction felt by individuals prompter to pick a choice out of several options follows an inverted U-shape. Namely, the optimal number of options to give individuals is not the maximum.

The intuition behind this is the following. When presented with an increasing number of options, the needs of people are more likely to be satisfied. In our work, more options means that Harambee’s young job seekers are more likely to find the skills that they are the most of proficient at among the ones we present them with. This abundance of choices can also be beneficial from a psychological point of view. In our case, it may comfort users into the idea that they have large sets of skills, and thus allow them to have a more positive approach in their job search. On the other hand, large sets of options have psychological and time costs. Studies have shown that our minds do not account for the full sets of available information when taking decisions, but only subsets (Wedel and Pieter, 2007). Following Reutskaja et al’s terminology, this “inattentional blindness” means that jobseekers presented with long lists of skills risk ignoring skills that they would have picked in smaller sets of options. In the case of Harambee users, a large set of options may also induce stress, especially linked to the sentiment of forgoing important skills. Worse, it may lead to a loss of motivation to answer the prompt correctly, and to users selecting skills by default. This evidence suggest that Harambee is right to worry about the number of skills presented to young jobseekers being too large.

Ordering of Skills

Once the lists of skills presented to young jobseekers have been narrowed down, the order in which the skills are presented is likely to impact their picks. As Harambee and the teams in Oxford are considering ordering the skills based on concepts such as transferability (essentially, the frequency with which a skill is associated with an occupation in ESCO), understanding how that order may influence users’ pick is critical. Intuitively, one could think that users tend to pick the first skill presented to them, especially when they must take an action (click “see more”, for instance) to see other options. Studies researching the effect of the order of questions in online surveys suggest that order does matter. A study conducted in the US in 2008 found that low-education respondents who answer surveys quickly – which is likely to be the case of Harambee users – a most prone to primacy effects (bias toward selecting earlier response choices). A study conducted in Xiao et al in 2024 also observes such bias, while finding that it is inexistent in “objective questions”, for instance those pertaining to demographics.

The implications for the Harambee platform are not straightforward. On the one hand, Harambee may seek to nudge job seekers into picking skills that Harambee’s counsellors consider as better picks. This would imply putting these skills on top of the list of options. On the other hand, Harambee may choose not to try and influence users’ picks. However, a random order would effectively lead to answers biased toward the first options presented. Countering this bias is not evident, which may be another argument to choose to hierarchize skills.

Nudging and Agency: the paradox behind libertarian paternalism is at the heart of the criticisms addressed to R. Thaler and C. Sunstein’s 2008’s book Nudge. Their argument is that modifying the architecture of choices – namely the way the information is presented to individuals bound to make decisions – does not influence the range of options an individual may pick, nor impair their capacity to choose freely. Moreover, the baseline architecture of choice tends to skew decisions toward bad outcomes. Namely, no architecture of choice is neutral, as the rationality of humans is bounded. Therefore, nudging does not impair agency.

Prompt

The evidence previously quoted finally suggest that choice overload is different from information overload. Namely, the sentiment of discomfort felt by individuals in choice overload does not come from the number of options only, but also from being prompted to choose a subset of options. Intuitively, the disproportion of the option set with the allowed number of picks – too many options, too few picks – makes choosing uncomfortable. This is likely to be especially true in the case of Harambee users selecting skills, as choosing skills to populate a CV is consequential. Being prompted to choose 5 skills is likely to create two issues: 1 – Encourage people to select 5 skills although they are not proficient in some of them. Picking less skills may make users lose confidence by suggesting they have few skills. 2 – Make users feel like they are not able to surface enough of their human capital, which may create frustration.

However, it is unrealistic to populate CV’s with more than 5 skills per occupation. Moreover, the occupation title also sends signals to employers about jobseekers’ skill sets. Therefore, we only recommend adjusting the prompt from “Pick the five skills that you are the most proficient at” to “Pick up to five skills that you are the most proficient at”.

All in all, evidence appear to justify narrowing down the lists of options presented to young jobseekers on the Harambee platform. It also gives evidence that the order of options matters, which may inform decisions taken by Harambee. Finally, it suggests choosing a prompt carefully, so as to not induce stress or demotivation on the platform’s users.

Bibliography

Neil Malhotra, Completion Time and Response Order Effects in Web Surveys, Public Opinion Quarterly, Volume 72, Issue 5, December 2008, Pages 914–934, https://doi.org/10.1093/poq/nfn050.

Reutskaja, E., Iyengar, S., Fasolo, B., & Misuraca, R. (2020). Cognitive and affective consequences of information and choice overload. In Routledge handbook of bounded rationality (pp. 625-636). Routledge.

Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.

Wedel, M., & Pieters, R. (Eds.) (2007). Visual Marketing: From attention to action. Hove: Psychology Press.

Xiao, Yaqing and Yan, Hongjun and Davidson, Erik, Response Order Biases in Economic Surveys (February 2, 2024). Available at SSRN: https://ssrn.com/abstract=3786894 or http://dx.doi.org/10.2139/ssrn.3786894.