Programme for the International Assessment of Adult Competencies (PIAAC), English Pilot Study on Non-Cognitive Skills

This online survey was designed to test the measurement properties of nine personality scales – the Big Five, Traditionalism, Self-Control, Self-Efficacy, Honesty/Integrity, Socio-Emotional Skills, Intellectual Curiosity, Job Orientation Preferences and Vocational Interests. Eight of these nine scales are existing scales (or combinations of scales) available for use in public domain. The scale assessing socio-emotional skills was developed by an expert group. The complete formulations of items from all the scales including the different forms/test conditions of the scales are presented in the data documentation.

Simplified scales
Based on the work of members of the expert group, simplified versions of original scales were developed (see data documentation). This was done in order to make the wording of the original scales more appropriate for use with general adult population (in many cases the original items were perceived as possibly too complex and abstract for less literate members of general population).
Not all items have simplified versions since in some cases the original formulations were seen by the expert group as suitable for the target population. This is especially the case in the Job Orientations, Integrity/Honesty and Vocational Interests’ scales as well as the entire Intellectual Curiosity scale. In total, there are 174 original items and 130 simplified or reversed versions of the items.

Neutral/middle point
In addition to comparing scales containing the original and simplified items, the second main design feature of this online survey was the use of a neutral/middle point in the Likert scales (agree/disagree). In particular, there were two versions of each original and simplified scale – one with 5 agree/disagree response options, including a “neither agree nor disagree” neutral/middle category and another with 4 response options, which did not include the option of “neither agree nor disagree”. This was done in order to see which of the two response formats worked better for each of the scales in target population.

Balanced scales
Some of the original scales were balanced and some unbalanced, with the majority of items being part of balanced scales. The unbalanced scales (self-control, self-efficacy and socio-emotional skills) were balanced by including alternative reverse formulations of a selected small group of items. The process of balancing was achieved by reversing the original formulations of 22 items (in the item bank document, the reversed items are marked with “R” while newly reversed items are marked as “new R”). These newly reversed items were tested against their original counterparts in order to see if the creation of balanced scales led to improvements (comparing both item- and scale-level properties).

Multiple choice vs forced choice
In case of the Vocational Interest scale, the two design features that were tested were the original vs simplified and multiple choice vs forced choice item formats. The original format of the Vocational Interest scale was multiple choice. However, the force choice format is often used in other Vocational Interest scales and the expert group wanted to test which of the two formats works better for general adult population.

Design of the online survey
The objectives of the online survey were the following, to test:
1. the measurement characteristics of the selected scales;
2. the relationships of the selected scales with background and other characteristics of respondents;
3. different item formulations – original vs. simplified;
4. different response options – with or without a neutral/middle category;
5. scales with different item formats – multiple choice vs. forced choice (voc. interests scale); and
6. the new balanced scales (in comparison with the original unbalanced scales).

Background questionnaire
The survey included a number of socio-demographic, economic and personal wellbeing indicators as well as a short cognitive ability test.
Socio-demographic characteristics: Gender, age, country of birth/residence, mother tongue, marital status, educational attainment, and parental education
Economic and wellbeing indicators: Broad activity status, occupational status, income, subjective health, social trust, life satisfaction, and personal wellbeing
Quality control questions: In order to check the quality of responses, the survey included three quality control items placed within the Big Five, Self-Control, and Socio-Emotional skills scales. These were used, together with other indicators of data quality, to create an overall quality control indicator and guided the exclusion of poor quality responses.