Harambee Panel
As one of our main tools to build a localized and inclusive ESCO taxonomy for South Africa, Tabiya organised panel discussion to channel Harambee's knowledge of South African young jobseekers.
The panel ended in May 2024 and its results are still being processed. This page is thus a work in progress.
Why a Panel ?
When it comes to assessing the transferability and credibility of skills acquired by job-seekers in the unseen economy, employers are likely to display strong biases. Namely, when a young job-seekers declares that he possesses skill A, an employers is likely to consider that skill A acquired in the unseen economy is not equivalent to skill A aquired in the seen economy, both in terms of the level of skillfulness (credibility) and nature of the skill (transferability).
To build an inclusive taxonomy, our challenge is, therefore, to try to get an unbiased overview of the skills that could be associated with unseen activities defined in the ICATUS framework. There exist no data systematically estimating the skillfulness of workers and people involved in the unseen economy, although some work has been conducted to assess the acquisition of basic skills (literacy, numeracy) among harambee users (cf. Kate Orkin). Our approach therefore relies on an attempt to oppose different biases in order to get a sense of the credibility of skills associated with unseen activities. Our guess is that, by making people who have different views on the unseen economy discuss, we may obtain a clearer picture of the value agents give to skills acquired in the unseen economy.
Participants to the Panel
Ideally, the panel would have included a dozen participants representing different interests. We expect members to represent:
Intermediation experts: Such as Harambee's employees, who work on the daily with both young jobseekers and employers. Their approach to skills is likely to be the most realistic, as it reflects both the will to improve the inclusivity or the South African labour market, and the realism acquired through their work with the main employers
Employers: Employers are likely to display negative biases regarding the transferability and credibility of skills acquired in the unseen economy. However, they are obviously the most important player when it comes to recruitment. Understanding their their take on the credibility of skills acquired o the unseen economy is essential to building an intermediation tool that successfully promotes the skills of job-seekers from outside any labour market (formal and informal).
Unseen economy workers looking for jobs: They are a precious source of information, not only because they are those who Tabiya and Harambee aim to help, but also because they have clear idea of what they are capable of. However, they have an evident incentive to report possessing as many skills as possible and a high degree of skillfulness in order to get hired. This upward biases is also likely gendered, as sociological studies have shown that women tend to understate their skillfulness, relative to men.
Unseen economy beneficiaries: They encompass all individuals that might benefit from a job-seekers involvement in unseen activities. This includes children, dependent and non-dependent adults, relatives operating household firms and receiving unpaid help from job-seekers... These participants are likely to have a positive take on skills acquired in the unseen economy, and play a critical role in changing narratives around them.
Mobilizing such a large and dispersed panel of individuals is complex and difficult to implement. Specifically, it was deemed close to impossible to mobilize employers to participate in such an exercise within a reasonable timeframe. Looking back after completing the exercise, it would have indeed been extremely difficult given the length of the exercise. As a result of the different constraints, the panel was held with 5 to 7 members of Harambee, the number of panels percipients varying between the phases of the exercise and in-between online surveys. While the limited panel participants made it difficult to aggregate answers, it made the exercise more manageable, especially when it came to ensuring that responses from panellists within the given timeframe and ensuring attendance at the online panel discussion. All that being said, the experience gathered during the Harambee panel may help us to bring to the table more diverse actors in the future.
Methodology of the Panel
The OMS research team’s initial proposal for the organisation of the panel was not possible to implement due to the lack of time of panellists and the overall length of the exercise. Adapting the exercise to the constraints gradually discovered by the team allowed it to come up with a more realistic methodology that is replicable, albeit improvable. This document presents the new methodology used after the initial conception of suggested skill lists by the OMS team.
Initial Training of Participants
Before starting the panel, the Tabiya team aimed to train panel participants in using concepts used to describe the signalling value of skills and experiences. To this end, participants were required to read a prompt on their own before starting to answer the online surveys. One may find the prompt below:
Because the forms were completed asynchronously and as a result of the large data on the survey resulting in technical issues, this phase of the process survey took a long time to complete. To facilitate panellists' work, it was decided midway to hold a two and a half hours meeting, during which the research team reiterated the concepts of sof signaling value and discussed examples with the participants before filling the surveys independently. This session was deemed useful by participants.
Online Surveys
Given that the initial plan of an in-person survey was quickly proved to be unrealistic, we decided to organise this survey asynchronously through Google forms to facilitate a more efficient live discussion online. This online Google form survey was made up of 6 sub-surveys covering all ICATUS activities under the three categories of focus for the unseen economy, which are, group 3: Unpaid domestic services for household and family members, group 4: Unpaid caregiving services for household and family members, and group 5: Unpaid volunteer, trainee and other unpaid work. Within each ICATUS category, we exclude activities that are considered “other”, for instance “other unpaid domestic service for household and family members” as the skills and activities are identical to the rest within each ICATUS category. In each of 6 sub-surveys, about 8 ICATUS activities were presented to participants, and for each the activities, all of the suggested skills were shown. To make the process easier and faster in the forms, the research team pre-organised the skills into bigger groups of similar skills, such as “communication skills” or “management skills”. For each skill, respondents were prompted to assign a signalling value score ranging from 0 “no signalling value” to 3 “high signalling value”. 1 meant that they deemed the signalling value to be low, while 2 meant that they deemed the signalling value to be medium.
Because the forms were completed asynchronously and as a result of the large data on the survey resulting in technical issues, this phase of the process survey took a long time to complete. To facilitate panellists' work, it was decided midway to hold a two and a half hours meeting, during which the research team reiterated the concepts of sof signaling value and discussed examples with theel participants before filling the surveys independently. This session was deemed useful by participants. By the end of this phase, which took place from April to May, the number of responses was deemed final for the first version of skills associated with ICATUS activities even though some panellists were not able to complete the full 6 forms within the given timeframe. The final number of answers was 7 for the first two surveys, 6 for surveys 3 and 4, and 6 for surveys 5 and 6. Although low, these numbers were deemed satisfactory for a piloting exercise and did not seem to hinder discussions.
Virtual Live Survey
As a final step of the exercise, the research team decided to organize an “in-person” panel on Teams on May 30th. The two- and half-hour session included all online survey participants except one. First, the research team re-introduced the stakes and concepts used in the exercise. Then, participants were asked to discuss the signalling value of skills the answers of which had a high variance during the online Google forms survey. Participants were asked to focus their discussions on likelihood, transferability, and proficiency at first, then open their discussions to other considerations. Finally, at the beginning of the last hour of the in-person panel, the participants were asked about their views on the exercise and potential improvements.
Panel participants were presented with a supporting booklet organised in the following way. Each ICATUS activities were described (including definition, tasks included, and tasks excluded) and accompanied by a temptative list of skills. Participants were then presented with the following prompt:
The Harambee in-person panel was the last step to associate final lists of ESCO skills to ICATUS activities, using their signalling value as a discriminating characteristic. It followed the online panel that took place between March and April 2024 and required panel participants from Harambee to score skills independently. As discussed previously, the in-person panel was meant to fulfil three goals. First, overcoming the issue of the high variance in answers for certain skills and occupations, which suggested misunderstandings rather than the diversity of opinions. Second, the panel was meant to elicit the underlying decision-making rules panel participants had been using to score skills. Namely, it aimed to highlight how signalling-value sub-concepts (likelihood, proficiency, transferability) were used by intermediation professionals in the South African context, and if other considerations entered their subjective signalling value scoring function. Third, the panel format was a first for Tabiya, and it thus served as a benchmark for future work on country-specific taxonomies of occupations and skills.
Results of the Panel
The Harambee panel was deemed successful overall, although it posed multiple technical and timeline issues. Not only did it make it possible to identify final lists of ESCO skills for ICATUS activities, but it also allowed the team to adapt the list of unseen activities to the South African context. Finally, it allowed the research team to disentangle the concepts used by labour market intermediation professionals to think about the signalling value of skills, some of which the teams had not previously identified. This experience was also a useful proof of concept for a qualitative approach to taxonomy building and suggests that reproducing this methodology in other national contexts should be fruitful.
Scoring High-Variance Skills
For each activity in ICATUS, skills were scored by up to 7 Harambee members based on their signalling value over the course of the online panel. However, aggregating these scores raised questions as to the relevance of majority rules or averaging the results. Given the low number of panel participants, summing scores or selecting scores with the most “votes” would have created spurious precision. It was thus decided by the team sum the results of skills for which the answers are similar. More precisely, we defined thresholds such that skills the answers of which display a higher variance shall be discussed again to avoid spurious precision. For each skill/occupation dyad, we computed the variance between the answers of participants. Then, for each survey we selected the skills with the highest variance to present them to panel participants again. In practice, this means that all skill/occupation dyad for which the answers were half 0 “no signalling value” and 3 “high signalling value” were included in the in-person panel.
During the panel, participants were prompted the discuss the concepts defined by our team (likelihood, proficiency, transferability) before agreeing on a final signalling value score. This exercise proved to be successful. It revealed that skills for which answers displayed a high variance were typically understood in different terms by participants, or that the way the signalling value concepts were used differed. For all these skills, the panel managed to agree on a final signalling-value score, which suggests at least a minimum agreement on an underlying signalling value function. To access further information on the distribution of signalling value scores and examples of skills disused in the panel, one may refer to [insert Tina’s doc name].
Learnings about subjective signalling value functions
Despite the initial instruction for the panel to use likelihood, transferability, and proficiency as the signaling value pillars to score skills by, there was frequent debate about other components that the panel included in the scoring. The first and most prevalent other component was South African context specific considerations. The panel often relied on their understanding of the South African labour market to assign signaling value scores. In fact, they relied on this as much (in frequency terms) as the other likelihood, transferability, and proficiency indicators. The other concept that frequently came up during the panel was the panel’s expertise on the expectation of the response of the average South African entry-level employer to a given skill. Given the Harambee panel’s labour market expertise, they were particularly well placed to infer how a representative employer would react to a skill. Thus, the panel used this inference in deciding which high-variance skills to assign no, low, medium, and high relative signaling value strengths to.
The virtual workshop provided the first opportunity for panellists to engage live with one another over the survey and skills selection process. Hence, two ad hoc decisions we made by the panel. First, the panel elected to remove the term “social service users” from ESCO skills. Due to the term’s ambiguity and incompatibility with the South African context, the panel elected to have it replaced with a more intuitive term such as “people” or “dependents”. The final decision implemented by the research team has been to replace ESCO skills comprised of the term ““social service users” with similar ESCO skills that do not include the term. For example, the ESCO skill “Assist social service users with physical disabilities” will be replaced with the ESCO skill “assist individuals with disabilities in community activities”. This rule has been adopted for uniformity in all skills in the localised taxonomy related to social service users.
In addition, the ICATUS Group Level activity entitled “Tending furnace, boiler, fireplace for heating and water supply” was removed from the localised taxonomy because of the inapplicability of the activity and its associated skills to the South African context. The closest version that could replace “Tending furnace, boiler, fireplace for heating and water supply” that is found in the South African Time Use Survey at the Group Level is “Chopping wood, lighting fire and heating water not for immediate cooking purposes”. Despite the existence of this potential replacement, the panel elected that this replacement was not worth including in the finalised taxonomy due to its likely inapplicability and scarcity in the South African labour market.
During the virtual workshop, the panel expressed that it found transferability, likelihood, and proficiency to be good signaling value indicators. They found that these indicators were especially enhanced with their knowledge of the South African context. The panel members cited lengthiness and repetitiveness as the main criticisms of the process. Adding page numbers to each survey was suggested as a way to manage survey length expectations and, in turn, better allocate time to completing the survey.
Types of arguments
a. Likelihood
The argument of the likelihood that a jobseeker has a skill was used very frequently during the discussions. The arguments relating to the likelihood of a jobseeker having a skill were based varied among examples. In most, they were based on the knowledge respondents have about the South African context. For instance, “teaching young horses” was deemed unlikely in link with the “daily pet care” activity, with respondents acknowledging that they would expect the likelihood to be higher in a European context. Another example is that driving skills were deemed to have a low likelihood in link with activities consisting in accompanying dependent adults or children, as participants stressed that moving around entailed less car rides than it would in Europe. In some other cases, the likelihood that a jobseeker has a skill based on an activity is deemed to be low because the skill tag is conceptually remote from the activity title. For instance, although a young jobseeker volunteering in a church could possibly participate in accounting activities, accounting skills seem to be conceptually to far remote from the idea an employer may have of volunteering in a church. Finally, respondents stressed that to answer they had to assume the exact activity young jobseekers were doing. For instance, volunteering in another household’s firm is not precise enough for an employer to infer what skills the jobseeker may have. This calls for a further improvement of the unseen taxonomy.
b. Transferability
Transferability was another of the main concepts presented to panel participants. Therefore, the concept was also widely used during the conversations. Sometimes, the concept was used in the right way. For instance, preserving food has been deemed to be weakly transferable, as professional food preservation relies on scientific principles and rules that one does not typically use in the unseen economy. However, the concept of transferability was also mixed up with other concepts multiple times. For instance, it was mixed up with likelihood, when participants stated that a given experience “did not transfer” to a skill. It was also used concomitantly with proficiency. When is skill was deemed to be basic, for instance tending grass, then it as deemed transferable. Overall, panel users seemed to deem skills to be on average more transferable than likely, but the two concepts appeared as equally important.
c. Proficiency
The concept of proficiency was almost not used by panel participants. More precisely, it was only invoked through the transferability of skills, when participants guessed that basic skills were more transferable and thus had more signalling value. Otherwise, discussions around how good a jobseeker might be able to get at given skills through given activities was not discussed.
d. Labor market demand
Discussions around labour demand were rather unwanted by the team prior to the panel, as the research team hoped that “what employers want” would not influence the making of skills lists for ICATUS activities. However, some interesting conversations happened where panellists stressed that some unseen experiences could have a very signalling value for very specific employers. They thus deemed it important to keep certain skills visible on our skills lists, to allow jobseekers to signal them. On the contrary, some skills were left aside as participants deemed the demand for it to be low.
e. South African context
As described before, the South African context was mobilised multiple times along with the concept of proficiency, usually to argue that a skill or occupation was not relevant to the South African context. The later was unexpected, as the panel decided to delete “Tending furnace, boiler, fireplace for heating and water supply” and “Accompanying non-dependent adults” from the list of unseen activities altogether. Considerations around the South-African context fed into discussions about other concepts such as likelyhood and transferability, for instance through qualifications. Indeed, some skills were deemed not to be transferable on the ground that they require qualifications in South Africa.Such an account for the South African context is perfectly aligned with what the research team hoped the panel would discuss.
f. Stretch between the activity title and the skill
Participants stressed multiple times that although skills relating to an activity might be likely and transferable, they are sometimes too far remote from each other. Therefore, an employer would be unlikely to think about skill A if a young jobseeker came with the unseen experience X. Conversely, some individual skills were deemed very valuable in themselves, but the panel thought that it is unlikely that a user picking the activity they are associated with would do it to pick them. Namely, some skills are unlikely to be picked, not because they are not valuable or transferable, but because picking them would require picking activities that users are unlikely to choose. This finding was of great relevance for the work of Tabiya, as it suggests that a taxonomy like ESCO may not be flexible enough for jobseekers to surface their human efficiently.
g. Naming of skills
Finally, multiple comments from participants referred to the naming of skills, that was deemed to make skills un-pickable for young jobseekers. In particular, many skills refer to “social services users”, for instance “assist social service users with physical disabilities” or “social service users to live at home”. Although the human capital signalled by these skill-tags was deemed to be relevant by panellists, they noted that the added precision of “social service users” made the skill less relevant, as it refers to a much more formal environment than the ones young jobseekers from the unseen economy are working in. In practice, skills like “assist social service users with physical disabilities” were deemed to have a low signalling value, although it was recognized that a skill like “assist individuals with physical disabilities” would have received a higher score. Yet, these skills do not have neutral equivalents that do not refer to “social service users”. Therefore, it was suggested to create new skills.
Takeaways about the signalling value function
The in-person panel proved useful to understand how Harambee workers think about the signalling value of skills in relation with occupations. Although we imposed three concepts (likelihood, proficiency, transferability), panellists appeared to find them useful, although proficiency was discussed less than the two others. Interesting remarks that feed into our concept of signalling value include the importance of qualifications (a skill may have a proficiency, transferability, and likelihood score but send a weak signal because it typically entails formal training). Second, it was interesting to see that skill tags influence the signalling value scores when they become too specific to a given context (social service users).
Learnings for future country-specific taxonomies
The in-person panel served as a proof of concept for further use of this qualitative methodology to adapt ESCO to local taxonomies. The exercise was deemed successful overall, albeit time-consuming. Unexpectedly, the exercise also brought about discussions that seem to call for solutions like Compass to overcome the rigidity of taxonomies.
Iterations of the panel exercise
Discussions surrounding the South African context, rather they touch upon what unseen activities young jobseekers might do or upon qualifications and demand side needs brought important insights used in building the “unseen” part of our localized taxonomy. Therefore, it gave encouraging results when it comes to using the same methodology in our next countries (France, Ethiopia…).
When it comes to organizing the panel, the research team ended up changing its initial plans multiple times. Online surveys proved to be very time consuming for respondents, who explicitly suggested that the research team shortens lists more than it did already to facilitate the work of panellists. The in-person panel on the other hand seems to have been liked by panellists and was fruitful. The suggestions for further panel iterations are the following:
1 – Apply the skills selections rules more stringently before presenting them to the panel, to reduce the time needed to answer the online surveys.
2 – Organise an in-person panel session to answer the first survey, along with the training session that was provided in this iteration. This was suggested by panellists so that the variance in answers is lower.
3 – Panellists should initially be asked which of the ICATUS activities they deem relevant for their local context, to avoid the situation of “tending furnaces”.
4 – Online surveys should include an indication of the overall length of the survey.
Compass
It is to be noted that this panel exercise suggested the need for a solution like Compass. Panellists stressed multiple time the lack of connection between skills and the ICATUS activities that they were linked with. While recognizing that some skills could have a high signalling value and be very relevant in the South African labour market, they considered that young jobseekers were unlikely to be able to highlight them because they were unlikely to pick certain ICATUS activities. This highlights multiple sources of rigidity in localized ESC0 taxonomies. First, our taxonomy is based on ICATUS, which covers all productive non-paid time uses but groups them in buckets that may not speak to platform users. Second, UX constraints call for the lists of skills associated to each ICATUS activity to be limited, which necessarily leaves aside certain skills. Therefore, the unseen taxonomy does not include all the skills that a young jobseeker might want to highlight, which is a common issue with ESC0 and results from the trade-off between a taxonomy that is inclusive (includes many skills for all occupations) and informative (limits the number of skills associated with each occupation). One solution to this is to have a flexible algorithm, allowing jobseekers to find skill through their experiences but also ad hoc, and thus highlight the full range of their human capital.
Online Survey Results
Six Google Forms surveys were administered to elicit a signalling value score from the Harambee panel members. This survey was filled in between the 15th of March 2024 and the 17th of May 2024. Once the survey completion deadline lapsed, we calibrated the answer “no signalling value” as zero, “low signalling value” as one, “medium signalling value” as two and “high signalling value” as three. This calibration allowed us to visualise each survey's distributions and easily quantify which skills sparked the most dissimilar answers from panellists (namely, which were the skills for which the answers had the highest variance). For analysis coherency, Surveys 1 and 2, which roughly (there were at most two ICATUS Group Level Activities that belong to ICATUS Division 3, 4 or 5 that were not found int their respective surveys due to survey capacity. These individual activities did not dramatically influence the shape of the distribution) represents ICATUS Major Division 3 were combined in our analysis. Similarly, Surveys 3 and 4, which roughly represent ICATUS Major Division 4, were combined, as were Surveys 5 and 6, which roughly represent ICATUS Major Division 5. This compartmentalisation is also done for simplicity since Surveys 1 and 2 received seven response submissions, Surveys 3 and 4 received six responses, and Surveys 5 and 6 received five responses.
Distributions
Each survey’s distribution is shown below. Each ICATUS Major Group has relatively different distributions. This is likely driven by the fact that the survey respondents’ sample size changed within each major division in an already small panel population pool. Note that the signal values are the aggregation of individual panel respondents’ answers. Therefore, Figure 1’s signal value axis is larger than Figures 2 and 3, which each had fewer respondents.
Variances
The variance of answers associated with each skill was particularly important because of their applicability to the live panel workshop discussion. The rationale used was that skills with the highest signaling score variances would be the most worthwhile to attain agreement on amongst all the survey respondents, given that this exercise could only be performed on a subset of all the skills chosen during the survey.
We isolated 20 of the skills with the highest variances per ICATUS Major Division. Ultimately, though, the skill with the 20th highest variance within each Major Division shared this variance with at least four other skills. Thus, the list of 20 high-variance skills expanded in each Division. A total of 26 skills ultimately comprised the high-variance skills for ICATUS Major Division 3, 24 for ICATUS Major Division 4, and 47 for ICATUS Major Division 5.
Following the virtual workshop, the high variance skills that were not discussed were assigned inferred signal scores based on the learnings from the panel. These scores were sent to three Harambee panel members for verification before their final incorporation into the taxonomy.
Virtual Group Discussion Results
A virtual two-and-a-half-hour workshop to discuss skills with the highest occurring variances per survey was hosted on the 30th of May 2024. Six of the seven survey respondents joined the session (though some respondents had to move into and out of the workshop). From this session, 12 skills from Surveys 1 and 2 (ICATUS Major Division 3) were discussed, nine skills from Surveys 3 and 4 (ICATUS Major Division 4) were discussed, and nine skills from Surveys 5 and 6 (ICATUS Major Division 5) were also discussed (this does not include skills and activities which need rewording or will be deleted from the taxonomy).
Of the 12 ICATUS Major Division 3 skills discussed, 83.3% were deemed to have no signaling value (a signaling score of zero), none of the skills were deemed to have low signaling value (a signaling score of one), 16.7% were deemed to have medium signaling value (a signaling score of two), and none of the skills discussed obtained a high signaling value. For ICATUS Major Division 4, 55.6% of the total ten skills were assigned a no signaling value score (a signaling score of zero), none were assigned a low signaling value (a signaling score of one), 22.2% were assigned a medium signaling value score (a signaling score of two), and 22.2% of the skills discussed obtained a high signalling value score. Finally, 66.6% of the nine ICATUS Major Division 5 obtained a no signaling value score (a signaling score of zero), and the remaining 33.3% was equally shared between low, medium, and high signaling value scores amongst the discussed skills. Figure 4 depicts the workshop’s signal value discussion over all ICATUS groups. It is clear from the figure that skills that were heavily contested in the survey were ultimately deemed to have no signalling value once panel members had a chance to pose arguments for and against these skills.
Final Skills Selection
Once we received the remaining high variance skills’ scores, we adopted a “traffic light” decision rule. That is, 33.3% of the skills with the lowest signaling value scores within an ICATUS Group (highest level of disaggregation) form part of the red category. These skills are removed from consideration on the SA Youth platform. 33.3% of the skills with the highest signaling value scores form part of the green category. These skills are surfaced at the top of the list of skills options that SA Youth users can select. The mid-range signaling value scores comprise the remaining 33.3% and are in the orange category. These orange category skills are surfaced beneath the full list of green category lists on the SA Youth platform. Below are three diagrams that show the absolute number of skills that will remain due to the enforcement of the traffic light decision rule for each ICATUS Major Division.
Last updated
