Harambee Panel
As one of our main tools to build a localized and inclusive ESCO taxonomy for South Africa, Tabiya organized a panel discussion to channel Harambee's knowledge of South African young jobseekers.
The panel ended in May 2024 and its results are still being processed. This page is thus a work in progress.
When it comes to assessing the transferability and credibility of skills acquired by job-seekers in the unseen economy, employers are likely to display strong biases. Namely, when a young job-seekers declares that he possesses skill A, an employers is likely to consider that skill A acquired in the unseen economy is not equivalent to skill A aquired in the seen economy, both in terms of the level of skillfulness (credibility) and nature of the skill (transferability).
Employers hold strong biases and priors when assessing the transferability and credibility of skills acquired by job-seekers in the unseen economy. That is, when a young jobseeker claims that he possesses skill A, an employer will likely note that skill A acquired in the unseen economy is not equivalent to skill A acquired in the seen economy, both in terms of substance (transferability) and intensity (credibility). This is a key challenge when building a taxonomy inclusive of unseen experience.
To build an inclusive taxonomy, our challenge is, therefore, to try to get an unbiased overview of the skills that could be associated with unseen activities defined in the ICATUS framework. There exist no data systematically estimating the skillfulness of workers and people involved in the unseen economy, although some work has been conducted to assess the acquisition of basic skills (literacy, numeracy) among Harambee users (cf. Kate Orkin). Our approach therefore relies on an attempt to oppose different biases in order to get a sense of the credibility of skills associated with unseen activities. Our guess is that, by making people who have different views on the unseen economy discuss, we may obtain a clearer picture of the value agents give to skills acquired in the unseen economy.
Panel Participants
Representation across 4 key interests/stakes in this space are ideal for a panel with these objectives:
Employers. Understanding the demand-side perceptions on credibility of skills acquired in the unseen economy is essential to building an intermediation tool that successfully promotes the skills of jobseekers coming from outside the labor market.
Intermediation experts who work with both supply (jobseekers) and demand (employers) sides reflect both the will to improve labor market inclusivity and realistic understanding of what works for job market success.
Jobseekers from the unseen economy can provide the clearest idea of what tasks they are capable of and to what extent. Additionally, they can also provide valuable insights on tendencies and upward biases in overreporting and misrepresenting skillfulness to get jobs.
Auxiliary beneficiaries of the unseen economy are all individuals that benefit from unseen work including children, dependent and non-dependent adults, families operating household firms etc. These participants can offer valuable third-party perspective on skills acquired in the unseen economy by jobseekers.
Mobilizing such a large and dispersed panel of individuals is complex and difficult to implement. Specifically, it was deemed close to impossible to mobilize employers to participate in such an exercise within a reasonable timeframe. Looking back after completing the exercise, it would have indeed been extremely difficult given the length of the exercise. As a result of the different constraints, the panel was held with 5 to 7 members of Harambee, the number of panels percipients varying between the phases of the exercise and in-between online surveys. While the limited panel participants made it difficult to aggregate answers, it made the exercise more manageable, especially when it came to ensuring that responses from panelists within the given timeframe and ensuring attendance at the online panel discussion. All that being said, the experience gathered during the Harambee panel may help us to bring to the table more diverse actors in the future.
Methodology of the Panel
The OMS research team’s initial proposal for the organization of the panel was not possible to implement due to the lack of time of panelists and the overall length of the exercise. Adapting the exercise to the constraints gradually discovered by the team allowed it to come up with a more realistic methodology that is replicable, albeit improvable. This document presents the new methodology used after the initial conception of suggested skill lists by the OMS team.
Participants were first briefed on key concepts to be used throughout the panel, followed by an online survey that aimed to elicit participants' perceptions/assessment of signaling values of skills under specific activities. The last phase was an in-person panel discussion focused on transferability and proficiency.
1. Concept briefing
In panels like these, that attempt to bridge theoretical interpretations of labor market behavior with practical understanding of the space, i.e. a collaboration of academics and practitioners, it is important that all parties precisely agree on the terminology and concepts being referenced. Panel participants were first provided a briefing on concepts used to describe the signaling value of skills and experiences, to be read before starting the online surveys.
2. Online Survey
The online survey had 6 modules covering all ICATUS activities under the three categories for unseen economy: Group 3: Unpaid domestic services for household and family members, Group 4: Unpaid caregiving services for household and family members, and Group 5: Unpaid volunteer, trainee and other unpaid work. In each module, , and for each the activities, all suggested skills were shown. For ease of use, the research team pre-organized the skills into categories similar skills, such as “communication skills” or “management skills”. For each skill, respondents were prompted to assign a signaling value score ranging from 0 “no signaling value” to 3 “high signaling value”. By the end of the exercise, a total of 7 responses were collected for the first two modules, 6 each for modules 3, 4, 5 and 6.
3. Virtual Panel Discussion
The Harambee in-person panel was the final step in associating a final list of ESCO skills to ICATUS activities, using signaling value as a discriminating characteristic. The two- and half-hour session included all online survey participants except one. For this discussion, the research team first identified skills that demonstrated high variance in signaling value based on the survey results. On the panel, participants were asked to discuss the signaling value of these high-variance skills, first focusing on on likelihood, transferability, and proficiency, before opening discussions into other considerations.
Results of the Panel
The Harambee panel helped achieve clarity on several aspects of this unseen skills-association exercise: first, it allowed the research team to adapt the list of unseen activities to the South African context; second, it allowed the research team to better understand concepts used by labor market intermediation professionals in thinking about the signaling value of skills. The exercise was also a useful proof of concept in combining qualitative inputs to taxonomy building and suggests that reproducing this methodology in other national contexts will be essential.
Online Survey Results
For each activity in ICATUS, skills mapped to it were scored by up to 7 Harambee participants on the online survey based on their perceived signaling value. There were two options in aggregating these scores: we could average the results or assign final value based on majority results. Given the small sample of responses, both of these approaches had high risks of generating "spurious precision". Therefore, the team opted for a segmented approach based on variance in perceived signaling value:
1) scores for skills-occupation dyads with no consensus (i.e. half 0 “no signaling value” and 3 “high signaling value”) were set aside for further deliberation through the in-person panel.
2) low variance skills-occupation dyads were assigned a final score that summed all responses.
Learnings about subjective signaling value functions
Despite the initial instruction for the panel to use likelihood, transferability, and proficiency as the signaling value pillars to score skills by, there was frequent debate about other components that the panel included in the scoring. The first and most prevalent other component was South African context specific considerations. The panel often relied on their understanding of the South African labor market to assign signaling value scores. In fact, they relied on this as much (in frequency terms) as the other likelihood, transferability, and proficiency indicators. The other concept that frequently came up during the panel was the panel’s expertise on the expectation of the response of the average South African entry-level employer to a given skill. Given the Harambee panel’s labour market expertise, they were particularly well placed to infer how a representative employer would react to a skill. Thus, the panel used this inference in deciding which high-variance skills to assign no, low, medium, and high relative signaling value strengths to.
The virtual workshop provided the first opportunity for panelists to engage live with one another over the survey and skills selection process. Hence, two ad hoc decisions we made by the panel. First, the panel elected to remove the term “social service users” from ESCO skills. Due to the term’s ambiguity and incompatibility with the South African context, the panel elected to have it replaced with a more intuitive term such as “people” or “dependents”. The final decision implemented by the research team has been to replace ESCO skills comprised of the term “social service users” with similar ESCO skills that do not include the term. For example, the ESCO skill “Assist social service users with physical disabilities” will be replaced with the ESCO skill “assist individuals with disabilities in community activities”. This rule has been adopted for uniformity in all skills in the localized taxonomy related to social service users.
In addition, the ICATUS Group Level activity entitled “Tending furnace, boiler, fireplace for heating and water supply” was removed from the localized taxonomy because of the inapplicability of the activity and its associated skills to the South African context. The closest version that could replace “Tending furnace, boiler, fireplace for heating and water supply” that is found in the South African Time Use Survey at the Group Level is “Chopping wood, lighting fire and heating water not for immediate cooking purposes”. Despite the existence of this potential replacement, the panel elected that this replacement was not worth including in the finalised taxonomy due to its likely inapplicability and scarcity in the South African labour market.
During the virtual workshop, the panel expressed that it found transferability, likelihood, and proficiency to be good signaling value indicators. They found that these indicators were especially enhanced with their knowledge of the South African context. The panel members cited lengthiness and repetitiveness as the main criticisms of the process. Adding page numbers to each survey was suggested as a way to manage survey length expectations and, in turn, better allocate time to completing the survey.
Types of arguments
a. Likelihood
The argument of the likelihood that a jobseeker has a skill was used very frequently during the discussions. The arguments relating to the likelihood of a jobseeker having a skill were based varied among examples. In most, they were based on the knowledge respondents have about the South African context. For instance, “teaching young horses” was deemed unlikely in link with the “daily pet care” activity, with respondents acknowledging that they would expect the likelihood to be higher in a European context. Another example is that driving skills were deemed to have a low likelihood in link with activities consisting in accompanying dependent adults or children, as participants stressed that moving around entailed less car rides than it would in Europe. In some other cases, the likelihood that a jobseeker has a skill based on an activity is deemed to be low because the skill tag is conceptually remote from the activity title. For instance, although a young jobseeker volunteering in a church could possibly participate in accounting activities, accounting skills seem to be conceptually to far remote from the idea an employer may have of volunteering in a church. Finally, respondents stressed that to answer they had to assume the exact activity young jobseekers were doing. For instance, volunteering in another household’s firm is not precise enough for an employer to infer what skills the jobseeker may have. This calls for a further improvement of the unseen taxonomy.
b. Transferability
Transferability was another of the main concepts presented to panel participants. Therefore, the concept was also widely used during the conversations. Sometimes, the concept was used in the right way. For instance, preserving food has been deemed to be weakly transferable, as professional food preservation relies on scientific principles and rules that one does not typically use in the unseen economy. However, the concept of transferability was also mixed up with other concepts multiple times. For instance, it was mixed up with likelihood, when participants stated that a given experience “did not transfer” to a skill. It was also used concomitantly with proficiency. When is skill was deemed to be basic, for instance tending grass, then it as deemed transferable. Overall, panel users seemed to deem skills to be on average more transferable than likely, but the two concepts appeared as equally important.
c. Proficiency
The concept of proficiency was almost not used by panel participants. More precisely, it was only invoked through the transferability of skills, when participants guessed that basic skills were more transferable and thus had more signalling value. Otherwise, discussions around how good a jobseeker might be able to get at given skills through given activities was not discussed.
d. Labor market demand
Discussions around labour demand were rather unwanted by the team prior to the panel, as the research team hoped that “what employers want” would not influence the making of skills lists for ICATUS activities. However, some interesting conversations happened where panellists stressed that some unseen experiences could have a very signalling value for very specific employers. They thus deemed it important to keep certain skills visible on our skills lists, to allow jobseekers to signal them. On the contrary, some skills were left aside as participants deemed the demand for it to be low.
e. South African context
As described before, the South African context was mobilised multiple times along with the concept of proficiency, usually to argue that a skill or occupation was not relevant to the South African context. The later was unexpected, as the panel decided to delete “Tending furnace, boiler, fireplace for heating and water supply” and “Accompanying non-dependent adults” from the list of unseen activities altogether. Considerations around the South-African context fed into discussions about other concepts such as likelyhood and transferability, for instance through qualifications. Indeed, some skills were deemed not to be transferable on the ground that they require qualifications in South Africa.Such an account for the South African context is perfectly aligned with what the research team hoped the panel would discuss.
f. Stretch between the activity title and the skill
Participants stressed multiple times that although skills relating to an activity might be likely and transferable, they are sometimes too far remote from each other. Therefore, an employer would be unlikely to think about skill A if a young jobseeker came with the unseen experience X. Conversely, some individual skills were deemed very valuable in themselves, but the panel thought that it is unlikely that a user picking the activity they are associated with would do it to pick them. Namely, some skills are unlikely to be picked, not because they are not valuable or transferable, but because picking them would require picking activities that users are unlikely to choose. This finding was of great relevance for the work of Tabiya, as it suggests that a taxonomy like ESCO may not be flexible enough for jobseekers to surface their human efficiently.
g. Naming of skills
Finally, multiple comments from participants referred to the naming of skills, that was deemed to make skills un-pickable for young jobseekers. In particular, many skills refer to “social services users”, for instance “assist social service users with physical disabilities” or “social service users to live at home”. Although the human capital signalled by these skill-tags was deemed to be relevant by panellists, they noted that the added precision of “social service users” made the skill less relevant, as it refers to a much more formal environment than the ones young jobseekers from the unseen economy are working in. In practice, skills like “assist social service users with physical disabilities” were deemed to have a low signalling value, although it was recognized that a skill like “assist individuals with physical disabilities” would have received a higher score. Yet, these skills do not have neutral equivalents that do not refer to “social service users”. Therefore, it was suggested to create new skills.
Takeaways about the signalling value function
The in-person panel proved useful to understand how Harambee workers think about the signalling value of skills in relation with occupations. Although we imposed three concepts (likelihood, proficiency, transferability), panellists appeared to find them useful, although proficiency was discussed less than the two others. Interesting remarks that feed into our concept of signalling value include the importance of qualifications (a skill may have a proficiency, transferability, and likelihood score but send a weak signal because it typically entails formal training). Second, it was interesting to see that skill tags influence the signalling value scores when they become too specific to a given context (social service users).
Learnings for future country-specific taxonomies
The in-person panel served as a proof of concept for further use of this qualitative methodology to adapt ESCO to local taxonomies. The exercise was deemed successful overall, albeit time-consuming. Unexpectedly, the exercise also brought about discussions that seem to call for solutions like Compass to overcome the rigidity of taxonomies.
Iterations of the panel exercise
Discussions surrounding the South African context, rather they touch upon what unseen activities young jobseekers might do or upon qualifications and demand side needs brought important insights used in building the “unseen” part of our localized taxonomy. Therefore, it gave encouraging results when it comes to using the same methodology in our next countries (France, Ethiopia…).
When it comes to organizing the panel, the research team ended up changing its initial plans multiple times. Online surveys proved to be very time consuming for respondents, who explicitly suggested that the research team shortens lists more than it did already to facilitate the work of panellists. The in-person panel on the other hand seems to have been liked by panellists and was fruitful. The suggestions for further panel iterations are the following:
1 – Apply the skills selections rules more stringently before presenting them to the panel, to reduce the time needed to answer the online surveys.
2 – Organise an in-person panel session to answer the first survey, along with the training session that was provided in this iteration. This was suggested by panellists so that the variance in answers is lower.
3 – Panellists should initially be asked which of the ICATUS activities they deem relevant for their local context, to avoid the situation of “tending furnaces”.
4 – Online surveys should include an indication of the overall length of the survey.
Compass
It is to be noted that this panel exercise suggested the need for a solution like Compass. Panellists stressed multiple time the lack of connection between skills and the ICATUS activities that they were linked with. While recognizing that some skills could have a high signalling value and be very relevant in the South African labour market, they considered that young jobseekers were unlikely to be able to highlight them because they were unlikely to pick certain ICATUS activities. This highlights multiple sources of rigidity in localized ESC0 taxonomies. First, our taxonomy is based on ICATUS, which covers all productive non-paid time uses but groups them in buckets that may not speak to platform users. Second, UX constraints call for the lists of skills associated to each ICATUS activity to be limited, which necessarily leaves aside certain skills. Therefore, the unseen taxonomy does not include all the skills that a young jobseeker might want to highlight, which is a common issue with ESC0 and results from the trade-off between a taxonomy that is inclusive (includes many skills for all occupations) and informative (limits the number of skills associated with each occupation). One solution to this is to have a flexible algorithm, allowing jobseekers to find skill through their experiences but also ad hoc, and thus highlight the full range of their human capital.
Online Survey Results
Distributions
Each survey’s distribution is shown below. Each ICATUS Major Group has relatively different distributions. This is likely driven by the fact that the survey respondents’ sample size changed within each major division in an already small panel population pool. Note that the signal values are the aggregation of individual panel respondents’ answers. Therefore, Figure 1’s signal value axis is larger than Figures 2 and 3, which each had fewer respondents.
Variances
The variance of answers associated with each skill was particularly important because of their applicability to the live panel workshop discussion. The rationale used was that skills with the highest signaling score variances would be the most worthwhile to attain agreement on amongst all the survey respondents, given that this exercise could only be performed on a subset of all the skills chosen during the survey.
We isolated 20 of the skills with the highest variances per ICATUS Major Division. Ultimately, though, the skill with the 20th highest variance within each Major Division shared this variance with at least four other skills. Thus, the list of 20 high-variance skills expanded in each Division. A total of 26 skills ultimately comprised the high-variance skills for ICATUS Major Division 3, 24 for ICATUS Major Division 4, and 47 for ICATUS Major Division 5.
Following the virtual workshop, the high variance skills that were not discussed were assigned inferred signal scores based on the learnings from the panel. These scores were sent to three Harambee panel members for verification before their final incorporation into the taxonomy.
Virtual Group Discussion Results
A virtual two-and-a-half-hour workshop to discuss skills with the highest occurring variances per survey was hosted on the 30th of May 2024. Six of the seven survey respondents joined the session (though some respondents had to move into and out of the workshop). From this session, 12 skills from Surveys 1 and 2 (ICATUS Major Division 3) were discussed, nine skills from Surveys 3 and 4 (ICATUS Major Division 4) were discussed, and nine skills from Surveys 5 and 6 (ICATUS Major Division 5) were also discussed (this does not include skills and activities which need rewording or will be deleted from the taxonomy).
Of the 12 ICATUS Major Division 3 skills discussed, 83.3% were deemed to have no signaling value (a signaling score of zero), none of the skills were deemed to have low signaling value (a signaling score of one), 16.7% were deemed to have medium signaling value (a signaling score of two), and none of the skills discussed obtained a high signaling value. For ICATUS Major Division 4, 55.6% of the total ten skills were assigned a no signaling value score (a signaling score of zero), none were assigned a low signaling value (a signaling score of one), 22.2% were assigned a medium signaling value score (a signaling score of two), and 22.2% of the skills discussed obtained a high signalling value score. Finally, 66.6% of the nine ICATUS Major Division 5 obtained a no signaling value score (a signaling score of zero), and the remaining 33.3% was equally shared between low, medium, and high signaling value scores amongst the discussed skills. Figure 4 depicts the workshop’s signal value discussion over all ICATUS groups. It is clear from the figure that skills that were heavily contested in the survey were ultimately deemed to have no signalling value once panel members had a chance to pose arguments for and against these skills.
Final Skills Selection
Once we received the remaining high variance skills’ scores, we adopted a “traffic light” decision rule. That is, 33.3% of the skills with the lowest signaling value scores within an ICATUS Group (highest level of disaggregation) form part of the red category. These skills are removed from consideration on the SA Youth platform. 33.3% of the skills with the highest signaling value scores form part of the green category. These skills are surfaced at the top of the list of skills options that SA Youth users can select. The mid-range signaling value scores comprise the remaining 33.3% and are in the orange category. These orange category skills are surfaced beneath the full list of green category lists on the SA Youth platform. Below are three diagrams that show the absolute number of skills that will remain due to the enforcement of the traffic light decision rule for each ICATUS Major Division.
Last updated
