Does Not Compute: Background and Methodology

Change the Equation (CTEq) engaged the American Institutes for Research (AIR) to analyze raw data from the 2012 Programme for the International Assessment of Adult Competencies (PIAAC), a household study conducted by the Organization of Economic Cooperation and Development (OECD) to assess “’key information processing competencies’ that are relevant to adults in many social contexts and work situations, and necessary for fully integrating and participating in the labour market, education and training, and social and civic life..” (OECD, 2013). AIR focused its analysis on results from test takers in the United States, where the study was conducted from 2011 to 2012 with a nationally-representative sample of 5,000 adults aged 16 to 64. (U.S. Department of Education, n.d.).

For CTEq’s study, CTEq and AIR analyzed results from PIAAC’s assessment of “Problem Solving in Technology Rich Environments” (PS-TRE), which the OECD defines as “using digital technology, communications tools, and networks to acquire and evaluate information, communicate with others, and perform practical tasks” (OECD, 2013). According to the U.S. Department of Education’s National Center for Education Statistics, PS-TRE:

assesses the cognitive processes of problem solving—goal setting, planning, selecting, evaluating, organizing, and communicating results. The environment in which PS-TRE assesses these processes is meant to reflect the reality that digital technology has revolutionized access to information and communication capabilities over the past decades.

In order to effectively operate in this environment, it is necessary to have mastery of foundational computer (ICT) skills, including (a) skills associated with manipulating input and output devices (e.g., the mouse, keyboard, and digital displays), (b) awareness of concepts and knowledge of how the environment is structured (e.g., files, folders, scrollbars, hyperlinks, and different types of menus or buttons), and (c) the ability to interact effectively with digital information (e.g., how to use commands such as save, delete, open, close, move, highlight, submit, and send). Such interaction involves familiarity with electronic texts, images, graphics and numerical data, as well as the ability to locate, evaluate, and critically judge the validity, accuracy, and appropriateness of accessed information. These skills constitute the core aspects of the PIAAC PS-TRE assessment. (U.S. Department of Education, n.d.)

Based on his or her responses to items on the PIAAC assessment, each test taker received a score between 0 and 500 points, which the OECD then used to define four levels to characterize test-takers’ proficiency or skill in solving problems in a technology rich environment: Below Level 1, Level 1, Level 2, and Level 3. CTEq and AIR analyzed PIAAC data  to estimate the number of individualsin each skill category and to describe characteristics of individuals, such as average earnings by skill level.   

In assigning proficiency levels, PIAAC excluded those individuals who were unable to take the PIAAC assessments because of language barriers or who refused to take the computer-based assessment used to measure these skills.   In addition, some respondents who indicated that they had no experience with computers were not asked to take the PS-TRE assessment, and others who began the assessment failed it after taking a series of core questions.   In estimating the percentage of people at each proficiency level, we assumed that individuals without computer skills or who failed the core test were had the lowest level of PS-TRE skills.  We excluded individuals who refused to take computer version of the PIAAC assessment and took a paper-based version instead, which disqualified them from taking the PS-TRE test.

To calculate the percentage and number of millennials who do not believe that a lack of computer skills has affected their chances of being hired, promoted, or given a raise, we excluded the small number of test takers who were not in the workforce because they were in school. PIAAC’s question reads, “Has a lack of computer skills affected your chances of being hired for a job or getting a promotion or pay raise?” PIAAC did not ask this question of people who indicated that they had no experience with computers on the job.

To determine average monthly earnings, by skill level, we used PIAAC data on monthly earnings, but excluded observations—about 1 percent of the sample—that indicated the individual earned more than $25,000 per month.  We then computed mean of the earnings variables across the four PS-TRE skill levels, without adjusting for education or other factors that are commonly associated with earnings.

To understand the relationship between earnings and PS-TRE skills, we also used a regression approach to examine how earnings vary with PS-TRE skills, while adjusting for age, gender, educational attainment, race, numeracy skills, and literacy skills.   These adjustments provide a summary measure of average wage changes across individuals who have different levels of PS-TRE skills but are otherwise similar in terms of personal characteristics that may be associated with earnings.  Using regression essentially allows us to ask how earnings change, on average, across skills levels once we account for these other characteristics.    

Sources:

OECD (2013), OECD Skills Outlook 2013: First Results from the Survey of Adult Skills, OECD Publishing Retrieved May 27, 2015, from http://piaacgateway.com/what-is-piaac/

U.S. Department of Education, National Center for Education Statistics, What is PIAAC 2012? (n.d.). Retrieved May 27, 2015, from https://nces.ed.gov/surveys/piaac/