Every time I get into a conversation about education lately, someone drops a side-comment about Finland, like it's the Valhalla of public schooling. In the past few months, the success of Finnish students on international exams has been widely touted and press-ganged into the service of various arguments about what we do and don't need more of in America. Most notably, perhaps, in the documentary Waiting for "Superman", Davis Guggenheim cites Finland's success as proof that you can have a good public school system without spending any more money per pupil than we do currently.
I don't know much about the Finnish school system—I'm sure it's a good one—but to give teachers, schools, and school-systems the sole blame and credit for differences in test scores is one of the most common misuses of data in the study of education. One should look first not at the schools but at the society that surrounds them: differences in the cultural and economic backgrounds of students and parents typically account for much more of the variation in test scores than do differences in pedagogy, school design, or teacher training. I spent the past week looking through the data on demographics, international test scores, and their interaction. The available data was copious, and it had some interesting stories to tell.
Background
Last year, 15-year-old students at randomly selected schools in 65 countries around the world sat for a series of internationally normed exams in reading, mathematics, and science, created and administered by the Programme for International Student Assessment (PISA). The results, published this year, showed the USA, as usual, doing worse than our status as the most powerful country in the world would seem to warrant. Leaving Hong-Kong and Shanghai out of the data—since they're not nations and probably aren't comparable[1]—we placed 15th in reading (tied with Poland and Iceland), 21st in science, and 29th in mathematics (tied with Portugal and Ireland); the latter represents a five-place drop from 2003, when we placed 24th in mathematics. Finland had the highest over-all scores of any nation. (See Appendix A for a full list of results.)
America's poor performance is especially stinging because we spend a hell of a lot on education—far more than Finland. From age 6 to age 15, the average student costs the US government and its citizens (public and private expenditure combined) $105,752 in cumulative schooling costs; for Finland that number is only $71,385. (These numbers are adjusted using purchasing power parities to account for differences in the cost of goods and services in each country.) By this measure, the US spends more on education than any other nation in the Organisation for Economic Coopration and Development (OECD)—i.e. probably any nation in the world—except Luxembourg (poor Luxembourg spends a stunning $155,624 per student from age 6 to 15 and still scores abysmally); of the 34 nations in the OECD, Finland is the 14th smallest spender. [2]
Mitigating Factors
A week or so ago, a commenter on this blog wrote, "I don't know what the Finns are doing right…." Well, part of what they're doing right is having very few disadvantaged students, low immigration rates, and a homogeneous national culture.
In the CIA World Factbook's list of nations by income inequality, Finland comes in 114th; the US, with greater income inequality than the Ivory Coast, Nicuragua, Cambodia, Thailand, Kenya, and every nation in Europe, comes in 42nd. PISA estimates that 10.4% of US students suffer from social, cultural, and economic (ESC) disadvantage, as opposed to only 3.9% of Finnish students.[3]
Perhaps even more important than its level economic and socio-cultural playing field is Finland's cultural homogeneity. 91.2% of Finnish citizens speak Finnish as a first language. 93.5% of them are ethnic Finns. 82.5% belong to the Lutheran church. Only 82.1% of Americans speak English as their native language. Just under 80% are "white," a broad, ill-defined, and permeable ethnic category. The largest religion in America, at 51.3% of the population, is Protestantism, really more a category of religions than a single religion. Almost 20% of American students are either first or second generation immigrants, as opposed to 2.6% of Finnish students.[4]
Thus Finnish schools face few of the challenges that American public schools must overcome. With fewer disadvantaged students, they likely have a lower incidence of serious learning deficits and behavioral issues; they have fewer students whose native language differs from that of the school; and most importantly, they have a shared sense of culture, history, and values, which must make the school—both as an extension of the Finnish state and as a collection of teachers and administrators—appear familiar and comprehensible to the parents. We can expect to find in Finnish schools none of the mistrust, miscommunication, and tribalism that plague inner-city schools here in America.
There are some interesting details relating to educational expenditure as well. First of all, America has an immense student population—the largest of any country that took the exam and more than two and a half times larger than the runner up.[5] A big student population means a lot more bureaucracy, and thus greater expenses for the same quality of education. Secondly, although the Finns spend less per pupil in absolute resources, they actually spend more
per pupil in terms of percentage of their GDP. Thus, though the purchasing power devoted to each student is less, the commitment of the available resources within the society is greater.[6]
Demographics and Test Scores
The people at PISA have been conscientious enough to analyze their data in terms of many of the demographic variables discussed above, allowing us to see the actual impact of those variables on test scores in each country. Unfortunately for our purposes, their analysis focuses on reading scores (new assessments with analysis focused on science and math will be carried out in the next five years), but presumably demographic impacts on test scores do not differ too much from subject to subject. (I will take a closer look at that assumption in my next post.)
According to PISA's analysis, adjusting scores to account for ESC (economic, social, and cultural) disadvantage[7] eliminates about 22% of the difference between Finnish and US scores. Using PISA's data, I calculated Finland's national average if they had as many immigrant students as the US and educated them as effectively as they educate their immigrant students currently; the result eliminated about 32% of the Finland-US score gap.[8] (NB: very likely, these are not cumulative effects; low-scoring immigrants in the US are mostly ESC disadvantaged as well, and accounting for both variables simultaneously would probably not eliminate more than about 32% of the score gap.)
Nationwide variables also account for some of the gap. According to PISA's analysis, adjusting scores to account for the number of students in the school system eliminates just under 20% of the score gap.[9] PISA provides no measure of cultural homogeneity and it is impossible to estimate the impact of that quality on national test scores, using their data. It's always hard to pin down culture in quantitative terms, but I suspect that if we could, it would account for a significant share of the US-Finnish test gap.
Even if we could explain away the entire gap, though, we still would have to account for the fact that we're spending more than twice as much per pupil as the Finns. My point is not that the differences in scores and spending can be entirely explained away, but rather that a direct simple comparison is impossible. Ultimately, the search for excuses is less productive than the search for explanations. As I looked deeper into this data, I grew less interested in whether or not the fault was in our school system and more interested in just what the data could teach us about the educational challenges that face our country. It can tell us a lot, I think, but this post is getting long, so I'll save that discussion for next week.
[1] ^ Presumably, Shanghai draws many of the best students from around China, through that nation's highly meritocratic system of public boarding schools, making it hard to compare their data to that of other countries. Hong Kong I know less about, but I'm hesitant to compare its results to those of entire nations.
[2] ^ PISA 2009 Results, Volume I, What Students Know and Can Do: Student Performance in Reading, Mathematics and Science. See table I.2.20. The PDF is viewable here, but you have to download the whole 272-page pdf, so you'll want a decent internet connection.
[3] ^ PISA uses a complex measure of socioeconomic advantage and disadvantage, called the PISA index of economic, social, and cultural status (ESCS). The ESCS index is calculated using a combination of parental job status and education, family possessions like cars and dishwashers that are indicative of wealth, and the availability of cultural and educational resources at home.
[4] ^ All these data are from the CIA World Factbook
[5] ^ See again PISA 2009 Results, Volume I, Table I.2.20
[6] ^ In the US, we spend 5.5% of our GDP on education; the Finns spend 5.4%. The 0-14 age-group, however, makes up only 16.4% of the Finnish population, as compared to 20.2% of the US population. If we imagine that educational expenditures are devoted entirely to 0-14 year olds, then the US and Finland spend approximately 27% and 33% respectively of their per-capita GDP per student. Of course, educational expenditures go to older kids as well, so those estimates are very, very rough. The website Trading Economics provides some good piecemeal data, summarized in the table below.
Per-Pupil Expenditure on Education as a Percentage of Per-Capita GDP at Various Levels of Education (2003-2007 average) | |||
Primary | Secondary | Tertiary | |
US | 21.5 | 24.57 | 24.92 |
Finland | 18.10 | 30.07 | 35.432 |
[7] ^ this adjustment is made based on the number of economically, socially, and culturally disadvantaged students, not based on the average economic, social, and cultural status of the nations. The number of disadvantaged students is estimated as the number of students who fall more than one standard deviation below the international mean on PISA's index of economic, social, and cultural status (see footnote 3).
The full analysis can be found in PISA 2009 Results, Volume II, Overcoming Social Background: Equity in Learning Opportunities and Outcomes, table II. Like volume I, it's a big file.
[8] ^ I did this calculation myself, because PISA's estimation appears to be either erroneous or based on a conglomeration of data from nations with very different immigrant situations. (In several nations outside the OECD—e.g. Dubai and Qatar—immigrants outscore native students by a wide margin; obviously, these are situations quite different from that abiding in the US.) The PISA analysis actually adjusts scores down to account for high immigrant populations, clearly an inappropriate adjustment for Finland and the US, where immigrant students consistently score below native students. You can find the PISA analysis in the second half of table I.2.20 of the PISA 2009 Results, Volume I, referenced above. My own analysis is based on the more detailed data found in table II.4.1 of The PISA 2009 Results, Volume II.
[9] ^ PISA 2009 Results, Volume I, Table II.2.20, again.
Appendix A: Average scores on the PISA exam by subject and country or city
|
|
|
- Countries marked with an asterix (*) are in the OECD; those without are not.
- Scores colored green are above the OECD average in that subject; scores marked in pink are below. Note that the OECD average is higher than the average for all 65 nations, because most non-OECD nations score well below most OECD nations.
Superbly written!
ReplyDeleteSo how does the US score in comparison with the Finns (or anyone else) once you factor out the ESC disadvantaged? Wouldn't that give us a "common denominator" and be a reasonable place to start thinking about our educational system? Without it, how do we even know what we're talking about?
ReplyDelete