Irwin Kirsch is the Director of the Center for Global Assessment at Educational Testing Service (ETS). He is also an expert on literacy and skills assessment. He has a wide experience directing US and international large-scale assessments on literacy. He is currently Project Manager of PIAAC (Programme for the International Assessment of Adult Competencies), an international large-scale assessment in which Portugal participates.
Skills and social inequalities
Observatory of Inequalities: Adult competencies – for instance, literacy or numeracy – determine the way people deal with daily demands and their ability to decode and interpret written or numeric language. Can we see it as being a major factor in creating social inequality?
Irwin Kirsch: I think, in countries like the United States, it’s fair to say that literacy has become a currency. So what’s happening in society is that literacy demands have changed over time, and what we expect people to be able to do with literacy and numeracy has continued to rise, as has the complexity of societies. We now expect more people to have higher levels of literacy. What we see in the data is that those people with better levels of skills also tend to be the ones who stay in school longer and get higher degrees. And as a result they have better access to the workplace and – on average – better incomes. So we see a relatively strong connection between where you are in the literacy or numeracy distribution and what you have access to from an economic point of view. We are also starting to see connections between where you are from an economic point of view and where you are in terms of social outcomes – whether or not you are living at or near poverty or whether or not you participate in civic activities and volunteering. There’s data now to show that there’s a connection between education, skills and health outcomes. So what we are finding is that skills turn out to be quite important and relevant not just to the opportunities you get economically, but also to the ones that play out socially. Here in the United States and in other places, you see gaps – large levels of inequality between those who have benefited and those who haven’t. In fact, here (in the United States) the gap is growing and has become very difficult to close. We wind up leaving segments of the population behind. As literacy takes on more importance, the consequences of not having skills become greater and the inequality is more prevalent.
OI: School attainment is a good predictor of income outcomes. But in Portugal and in many OECD (Organisation for Economic Co-operation and Development) countries, there are big differences among people having similar levels of school attainment, namely those who concluded tertiary education. Do you think those income differences can be partially explained by the skills distribution?
IK: I think what you find is that for any level of education in a country – let’s take the United States or take Portugal – if you look at the people who completed ninth grade, 12th grade or even a level of college, you’ll see a wide variation within each level of schooling in terms of the skill distribution. What the data seem to suggest is that even within a particular level of education, those with better skills on average do better than those with lower levels in terms of labour force participation and the kinds of jobs that they have access to – and, therefore, the wages that they get. That varies across countries, but in countries like the United States, that’s a significant factor. Not surprisingly, what is important is not just the number of years you spend in school but the knowledge and skills you acquire.
OI: To what extent do people’s competencies depend on their social background? Can you identify the OECD countries where inherited adult competencies are more and less pronounced?
IK: I think background does make a difference. Parents have a significant impact on their children both in terms of the genes that are passed on to them and in terms of the environment they provide. Some background characteristics are important because they correlate with outcomes that are seen as important. But, often times we need to be careful how we interpret these findings. For example, in the United States, the minority populations tend to be disproportionally represented at the lower education levels and lower skill levels. This is the “achievement gap,” which typically gets defined in terms of race and ethnicity variables. But what is really behind the gap is poverty and other characteristics. It is not race and ethnicity that cause the inequality; it’s the fact that a disproportionate number of minority children come from poor backgrounds. It is important, I think, to understand what variables are simply covariates versus what variables actually contribute, from a causal point of view, to outcomes of importance. For example, we see in the PISA (Programme for International Study Assessment) data that the social backgrounds of parents have a significant impact on reading proficiency. But we found in the PISA 2000 survey that kids on average who came from low socioeconomic backgrounds but were highly engaged in the practice of reading actually scored on average better than kids who came from high socioeconomic backgrounds but with low engagement in reading. You do find data that suggest that you can overcome some of these background characteristics, but it’s difficult, and other characteristics have to be in place.
OI: But isn’t that kind of engagement quite related to family background?
IK: Sure, but you do find, as I was saying, parents from low socioeconomic backgrounds whose children are highly engaged readers. If you think of engagement in terms of motivation, behaviours, and values, then it makes sense that children who are highly engaged can overcome some background characteristics. I think it’s important to begin to understand the relationship between those variables and achievement and then figure out, “How am I going to do an intervention? How do I do an intervention where I can bring about those kinds of changes?”
OI: As it was said in the conference (International Large-Scale Assessment Conference. The Role of Large-Scale Assessments in Educational Policy) it’s important to debate school policies…
IK: Right, but there are also home values and home practices that also make a difference. While we always tend to focus on the schools, we have to remember that kids spend more time outside of school, and I think that in today’s world, as societies are becoming more complex, individuals are going to have to take more responsibility for their own lives and their own “education.” We need to recognize that a lot of learning takes place outside of the schools, and so in order for that to happen, people need to develop the skills, the practices and the values they will need to become independent lifelong learners. Schools won’t be the sole source of learning, because of the technology, the Internet…
OI: And the skills you learn or develop in the labour market are also very important…
IK: Sure. What we find here and in other places like Canada, the United Kingdom and some other countries is that those with better skills tend to stay in school longer. And those who stay in school longer earn higher degrees, and get access to jobs where those skills are also required. So, they continue to develop their knowledge and skills through their job. And, the data show that these same individuals also have access to training programs that are provided by their employers. The result is a cycle in which those who have acquired skills have more opportunity to use those skills both to reinforce what they already know and can do as well as to develop new knowledge and skills. This continuous engagement also helps as one grows older. Cognitive aging seems to be a fact of life, yet the data seem to suggest that those who are more engaged – that work, for example – tend to have less steep declines. It takes a much longer period of time before they lose those skills. If you compare skills to physical activity, you find that people who work out and are more physically active tend to keep muscle tone and stay fitter longer. This is because even though their body is aging, the regular exercise they do helps to keep them fit. So if you think about cognitive skills from a practice or exercise point of view, it makes perfect sense.
OI: Analytically, we can identify several types of competencies: numeracy, literacy, problem solving… Do the individual competencies tend to have a systemic nature, based on multidimensional average skills, or do the individual competencies vary according to each specific dimension?
IK: While I think there are general competencies that cut across various domains, I also feel there are some unique characteristics that must also be learned. In my opinion it is important to understand both the general as well as the unique characteristics. For example, when we look at reading literacy, we can analyze it along a single scale or as a set of subscales. For example, in PISA reading literacy is treated as a single scale for many analyses because the subscales have pretty high correlations. And for some purposes this is fine; however, when you start to look at subgroups in a population, when you compare boys and girls or men and women, or when you compare different subgroups within a country by, let’s say, race and ethnicity – you see some significant differences. If you are an economist – as we heard at the conference – then the aggregate data are fine because the correlations and the rank ordering of people are pretty similar. But if you are interested from an educational point of view or from a policy point of view, you can look at subgroups on various subscales and find very interesting differences and these differences have implications for developing interventions. I’ll give you an example: If you teach reading or literacy only as connected discourse, then you ignore the fact that much of the information that we process as adults is not presented in paragraphs or as prose. It’s in tables, it’s in charts, and it’s in documents. My view is that you cannot assume that someone who can read prose can also read documents. There are things that are unique in the structure of documents that ought to be taught. They not only help you from a quantitative point of view, in terms of the math, but they also help you from an information processing point of view, because information is organized in documents differently than it is organized in prose. These different organizational principles can be and should be taught using different strategies. Similarly, now that technology has come into play, people say the same thing about digital text or electronic reading. Some will argue, “It’s just reading, you don’t need to teach people how to read digital text because they know how to read printed text. It’s the same set of skills.” There are other people, myself included, that say, “Well, yes, there are some sets of skills that are transferable across these two modes – printed and digital – but there are some things that are different in the digital environment that are important. While some kids and adults would figure it out on their own, you can make it much more effective and efficient if you understand and teach what is different.”Challenges related to human capital
OI: To what extent do you think politicians and policy makers are aware of the strategic relevance of human capital? Barack Obama, for instance, recently highlighted this question. But don’t you think that the hardcore political discourse on economic development strategies tends to rely on other aspects, such as tax policy, labour law or financial policies?
IK: I think people like President Obama and others have recognized that education is no longer an option, it’s a requirement, and it’s a responsibility, but it´s occurring in a time when countries are facing severe budget problems. So making new investments in education is difficult. People know that if you don’t, it’s a problem, but there’s also a problem finding the money to do it and then getting everybody to recognize that they have to invest in themselves. So it’s not just the government making the investment. People have to recognize that they have to develop human capital for themselves. They have to be willing to make the investment.
OI: But government plays a major role in encouraging people to invest in themselves…
IK: I agree and I think that in countries like the United States, governments ought to create incentives for individuals to want to do that. I think we will find that it is good for individuals and good for societies. It is important that we reduce high levels of inequality that exist in many countries including the United States. So I think what we are going to find in the future is that healthier societies – and I don’t mean only physical health – will be the ones that have lower levels of inequality and higher levels of skills.
OI: Some authors have talked about “clashes” between societies, whether they are cultural, political or ideological. Do you think that, in a globalized world, OECD countries will face a “literacy clash” with countries like China or India? IK: There’s a newspaper editorialist here in the United States named Thomas Friedman who works for the New York Times. He makes the point, which I think is quite valid, that in years past, it used to be the nations that were important, and corporations operated within national boundaries. However, in a global world, the national boundaries kind of dissolve and the corporations take on more importance. Most large corporations are spread out and are global, and the income that they derive and the labour force that they use are spread out around the world. So he argues that national boundaries give way to corporate boundaries, and now corporate boundaries are giving way to individuals. So it’s the individuals who will compete with each other; it won’t be countries that will compete with each other. In knowledge economies it will be individuals and corporations that will compete for ideas and talent. His point of view is that we need to educate more people and we need to educate them better. Societies need people at the top to drive innovation, but they also need people to be lifelong learners because technology and information is changing so quickly, and then those people need to able to work across national boundaries. The people and corporations who can do that best will be the ones who benefit the most.
OI: But the countries that better host and attract those corporations and qualified workers will also be the ones that benefit the most…
IK: Sure. But then you have to look at the structure of the labour force within countries. What kinds of jobs are available and what kinds of wages do these jobs pay? Because that comes back to the inequality question. If you have a society primarily driven by service-level jobs, where the wages are depressed and only a relatively few jobs are at the top, then you will, by default, have high levels of inequality. The question is how do you produce a labour force with skills that create jobs that pay a livable wage so there will be less inequality? I think that one of the big issues over the next period of years is going to be what the role of government is. How much of the prosperity of a country will get shared and how does income and opportunity get distributed? That will make a difference in terms of the quality of life and the well-being in that society, but also the kinds of institutions they have to develop to support that kind of society. I think that’s going to be a really interesting debate. Countries like the United States are struggling with those issues right now.
Understanding and using assessment data properly
OI: The data collected in these international assessment surveys can be used, at least indirectly, to evaluate national education systems?
IK: I think the databases are useful for some purposes. For others they have less utility. Over time, I think assessments are responsive to what policy makers want to know, what researchers want to know. I see databases like IALS (International Adult Literacy Survey) and PIAAC (Programme for the International Assessment of Adult Competencies) being particularly useful because they not only show what the recent product of the schools looks like and what they are able to do, but they have what I would describe as “concurrent validity studies.” They contain background questionnaires that gather data around immigration, education, labour market experience, income, lifelong learning, health and so much more. These data along with the direct assessment of skills provides a warehouse full of information about how human capital is distributed within and across countries as well as how these skills relate to important outcomes including education, labour force participation and social engagement. Policy makers can ask, how is a given level of skill related to educational attainment? Given my level of skill, how is that related to access to the workforce? How is that related to growth of skills over time? What is the role of technology in people’s lives? Are people disadvantaged economically or socially if they don’t have access to technology or aren’t technologically literate? So you can begin to ask richer kinds of questions about society and outcomes. You heard Eric A. Hanushek (PDF presentation, vídeo) say that 10, 15, 20 years ago, economists didn’t have skill data. What they really had was years of schooling, and when you look at years of schooling and you plot it against GDP (gross domestic product), you don’t see much of a relationship. Now that they have skill data, they see a much stronger relationship to economic development or GDP growth. Other people are now showing that it is not just related to GDP growth but also to well-being and to democratic institutions. As more people have more access to this kind of data, and more of that data becomes available, I think more and more questions will be asked, more and more questions will be addressed, and policy makers will have more information to make decisions and make investments.
OI: But to what extent do you think these skill assessments can evaluate the reliability of national education systems?
IK: I think the information they provide improves over time as more and more scholars from different disciplines become more interested in the studies and, along with policy makers, ask for these studies to address more questions. I don’t think any single survey is designed to answer all questions. I think it is important that people don’t over-interpret the data and use the data in ways that are appropriate – that is, the inferences that they make are appropriate for the data collected – then these surveys will continue to be quite useful and quite informative. So there is this give-and-take: You have people who produce the data, you have people that will criticize the use of the data, and people who will interpret it to make policy decisions. It’s like a cycle, and from my perspective you want each cycle to be better than the one before. You want people to be using the data better, you want the data to become better, and you want people who are looking at the data to make sure that people are using the data appropriately.
PIAAC: a new generation of skill assessment surveys
OI: School attainment was, until 20-30 years ago, the best available tool to measure people’s competencies. Nowadays, we have much more accurate instruments to do it. Can you please give me a brief overview of the type of competencies that are going to be assessed in PIAAC, and the way PIAAC might overcome past assessments like IALS?
IK: I think PIAAC represents the next generation of assessments. I think IALS and ALL (Adult Literacy and Lifeskills survey) were innovative for the time and did a lot to get economists and others to understand the importance of skills as opposed to just the credential of education or the years of schooling. But they left some things on the table. For example, we didn’t really measure people at the very low end and we didn’t extend the upper end very far. In PIAAC we want to do a couple of things: we want to understand both ends a little bit better and we also recognize that in the adult environment, it’s difficult to not have access to or use technology. So we also want to measure something about how well people could use technological environments, like the web or email, to solve everyday kinds of problems that they might have.
The skill domains that exist in PIAAC are literacy and numeracy, and these are very similar to what was measured in the past. The only difference is that most people in PIAAC will use the computer, and because we can do it on the computer, we can broaden the construct of literacy to include electronic reading tasks. And, in numeracy we can do additional things that we couldn’t do with paper and pencil, and we can capture information besides just “right” and “wrong,” so that will help improve the measurement. But, in addition to that, we also are including what we would call “component skills,” that is, measurement of vocabulary, measurement of sentence fluency and paragraph comprehension. That will tell us whether or not people with low levels of skills are lacking in some basic “mechanical skills” that you need in order to be able to read and use print effectively and efficiently. We will also be able to say how many people use and have access to technology and how their technological skills are distributed with respect to problem solving.
Here’s what I think the data will show us: There are segments of the population in each country for whom components skills are not very strong, and for those people their literacy skills are also not going to be very strong. So we will find that components skills are necessary for developing good literacy and numeracy skills, but they are not sufficient. In addition, I think we will find that there are people who have mastered the components but still struggle with reading literacy and numeracy skills. These individuals could benefit from additional education or training to help them improve in these basic skills. This will be important because unless individuals have at least a moderate level of competency in literacy and numeracy, they are not going to be able to deal with problem-solving skills, because they are more complicated on average. So policy makers will find that literacy and numeracy skills put an upper bound on these more complicated problem-solving skills. If individuals don’t have good levels of literacy and numeracy, it’s almost impossible for them to be able to have good problem-solving skills. And if you don’t have good component skills, you can’t have good literacy and numeracy skills.
So what will come out of PIAAC , on the one hand, will be what the distributions of human capital look like in a country. We will then measure whether these relationships exist and how they exist in each country. But then we will also see how these things relate to outcomes and how similar and different these relationships are across the countries. So we will have a much better understanding, not just of the distribution of human capital, but also how it is connected to various outcomes. I think PIAAC will be a very rich source of information for participating countries in terms of the kinds of questions they can ask and at least begin to address them from a policy point of view.
Originally published in Observatory of Inequalities, 2011
