Sunday, January 9, 2011

Finland (Part 2)

Last week, I posted an article about the demographic causes of Finland’s preeminence on the 2009 Programme for International Student Assessments (PISA) exams. My purpose in that post was to show that a simple comparison between countries is impossible—but this is not to say that we can learn nothing from international comparisons. We can learn a lot, but only through careful and detailed examination of the data. I’ve spent much of the past two weeks sifting through that data, and I’ve come up with some interesting leads.

All of the issues addressed here bear further research; in general, it's dangrous to try to draw causal inferences from survey data—I can only hope that some better-funded researcher will look deeper into some of these issues.

1.       Rich kids are doing fine.
Breaking the population of each country down into quarters, according to economic, social, and cultural (ESC) status provides some interesting insight. First of all, the most ESC-advantaged quarter of the US population is actually quite competitive with the most advantaged quarters of other nations: our most advantaged quarter scores 8th in the OECD in reading (compared to 14th for all four quarters combined), and only one country (New Zealand, oddly enough) has a most advantaged quarter that scores statistically significantly higher than ours.[1]
Data on math and science scores is much more limited, but what data I have suggests that our top ESC quarter isn’t doing quite as fine in those subjects; but it is definitely closer to fine than the rest of the country. My very rough estimates put the mathematics and science performance of our most advantaged quarter around 16th and 10th in the OECD, respectively, compared to 25th and 17th for all four quarters combined.[2]

2.       Poor kids and lower-middle-class kids aren’t doing fine, but for very different reasons.
This is interesting—and completely unexpected. Simple demographics seem to account for the weak performance of the most ESC-disadvantaged quarter of the US population. Our most disadvantaged quarter is highly disadvantaged, considering how wealthy we are overall; it ranks 20th in average ESC status within the OECD. In its mean score on the PISA test, however, it is tied for 18th—so, in fact, our bottom ESC quarter is outperforming the less impoverished bottom ESC quarters of a couple other countries. Now, we shouldn’t be too proud of that, considering our high expenditure on education, but it’s a very different situation from that of our lower-middle-class.[3]
The second quarter of the US population in terms of ESC status—the quarter second from the bottom, that is—is not so bad off, in terms of ESC status, by PISA’s measurements: it’s the 11th most ESC-advantaged second quarter in the OECD—yet it ranks 21st in terms of scores on the PISA exam. Lacking a demographic explanation, we might attribute this poor performance to a failure on the part of the public schools available to these students—although, it’s also possible that there are cultural explanations, e.g. attitudes towards education—that are not easily captured in demographic data.[4] More>>

3.       Separate is not equal.
Schools in most nations are segregated by class, and US schools are more segregated than most.[5] According to PISA’s data, schools serving disadvantaged kids are significantly worse places to learn, no matter what your individual ESC status; this too is true in nearly every country in the OECD, and it is less true in the US than in most, but it is still an important reality of education in today’s industrialized nations.[6]
To illustrate the point—and I hesitate to illustrate the point, for it has some disturbing implications—suppose that two students of identical ESC status—say they are both in the 75th percentile of ESC status in the US—attend different schools, one advantaged the other disadvantaged. On average, the student attending the advantaged school scores 62.6 points higher on the PISA reading exam than the one attending the disadvantaged school. That score difference is close to twice the US-Finland score gap in reading.[7]
                The difference in education quality between advantaged and disadvantaged schools has at least two sources. One is the quality of the resources available to the school, but I suspect the more significant factor is the one that Justice Warren sites in Brown v. Board of ed.—that is, the inequality inherent in a system that is segregated by race or class. A school full of students from disadvantaged homes concentrates the academic and behavioral difficulties that arise from underprivilege, so that each student suffers not only his own disadvantages but the combined disadvantages of all his peers.

4.       Education begets education.
The US ranks third in the OECD in terms of per-capita GDP,[8] making it one of the wealthiest nations in the world—yet, it ranks 10th in terms of average ESC status.[9] Here’s why: PISA measures ESC status by looking not merely at measures of family wealth, but at measures of social and cultural status as well—parental education levels, educational resources available at home, and cultural possessions, such as classical literature, poetry, and works of art. The result is a more useful measure, because educational outcomes are more closely correlated to parental education levels and home culture than to income.
Thus, over the long term, a lackluster school system leads to lower average ESC status within the society, which leads in turn to worse educational outcomes in the next generation. Similarly, improvements in a school system, if they are sustained, create a snowball effect of rising outcomes, with each generation of parents providing better at-home support than the last.

5.       If you have kids and live in the USA, don’t get divorced.
The gloomy joke in that headline is that we don’t know, of course, whether the relationship between single parenthood and low-achievement is causal. The data seems quite clear, however, that being from a single-parent family is bad luck for a kid, more so in the US than almost anywhere else in the world.
The negative relationship between single-parent homes and poor test performance in the US holds even after you account for a host of other family variables. Two students in the US, both speaking English as a first language, with no immigrant status, identical parental educational attainment and occupational status, the same number of books and other cultural possessions at home, and the same amount of family wealth, but differing crucially in the number of parents at home will score, on average, 18 points apart on the PISA exam. Only three nations that took the PISA exam—Jordan, Qatar, and Dubai—have a stronger correlation between single parent family status and educational performance.[10]

6.       …And don’t buy TVs, either.
Again, this isn’t necessarily causal—I haven’t even run any regressions on it—but the negative correlation between a US student’s performance on the PISA exam and the number of TVs that her family owns is large—and consistent across all three subject tests. 78.86% of US students report having three or more televisions at home; that’s the highest percentage of three-TV students in the OECD. In the US, those three-TV kids score lower than students who report owning only one TV by an average of 41 points on the reading exam, 34 points on the math exam, and 40 points on the science exam, making US students one of the four or five most television-addled in the OECD.[11] (Again, no causal relationship between TV and test-scores has been demonstrated, only a correlation, so "television-addled" isn't quite the right phrase here. It’s possible that families that own more televisions tend to be less intellectual and thus produce weaker students without any help from the TVs… My money’s on the TVs though.) More>>


More About Rich and Poor Kids

It’s informative, I think, to look at the data for all four quarters of the US and Finnish population by ESC status.



Bottom Quarter
Second Quarter
Third Quarter
Top Quarter
Reading Performance
ESC Status
Reading Performance
ESC Status
Reading Performance
ESC Status
Reading Performance
ESC Status
Mean Score
Rank
In the
OECD
Rank
In the
OECD*
Mean Score
Rank in the OECD
Rank in the OECD*
Mean Score
Rank in the OECD
Rank in the OECD*
Mean Score
Rank in the OECD
Rank in the OECD*
Finland
504
1st
5th
257
2nd
4th
548
2nd
4th
565
4th
10th
US
451
18th
21st
481
21st
11th
512
14th
11th
558
8th
10th
US-Finn score gap
53
46
36
7**

* In descending order of ESC status: thus, for example, the 5th in this column for Finland in the Bottom Quarter section means that the most ESC-disadvantaged quarter of Finland has the 5th highest mean ESC status of any such quarter in the OECD.
**not statistically significant, even at very low confidence levels.

It's worth mentioning a hidden caveat in this data, though the significance of that caveat is difficult to assess: the PISA exams are administered to a randomized sample of 15-year-old students in each country—not represented, it seems, are those who drop out of school or switch to homeschooling before the age of 15. Dropouts presumably tend to be those members of the bottom ESC quartile with the worst educational outcomes; by selecting them out of the data, we therefore may be inflating the score of the bottom quarter. Most students who stop attending school do so between the ages of 16 and 18—in many states its illegal to do so earlier, which doesn't mean it doesn't happen—so it's difficult to find data on dropout rates among younger students. For the same reasons, though, we can probably expect these dropout rates to be relatively small. Lacking data on either pre-16 dropout rates or the demographics of such dropouts, its impossible to speculate further on the impact of excluding these kids from the data.


More about TVs

American kids with only one TV at home score an average of 33 points above the national average, across all three subject exams, and place only 11th in the OECD, when compared to one-TV kids from other countries; overall, we placed 20th, as a nation. Other big winners in the one-TV comparison are Australia and Luxembourg, which jump to 1st and 18th from 6th and 29th, respectively.

In general, correlation between TV ownership and test-scores varies interestingly across nations, and I encourage the reader to look at Table 2 below, which provides a summary of the available data. In a few countries—Turkey, Mexico, and Chile, for example—television ownership is presumably a sign of wealth, and is therefore associated with higher test scores. It’s hard to know the extent of this wealth-indicator effect, but it seems likely that many OECD countries are sufficiently wealthy to render it irrelevant. In the table below, I’ve included the percentage of each country’s student population that has three or more TVs at home and the per-capita GDP for each country; both of which provide some hint of the extent to which television ownership may be a predictor of wealth in a given country—i.e. in countries with higher GDP and more TVs, TV-ownership is less likely to be an indicator of wealth.

The exception would appear to be Israel (see the bottom of the table below) where television-ownership and high performance on the PISA are strongly and positively correlated, despite a high per-capita GDP. I asked an Israeli friend about it and he offered the following explanation, which I find compelling and kind of fascinating. People in most countries turn on the tube to watch either domestically produced television or foreign—usually American—television programming dubbed into the national language. In Israel, most television is American or British programming, in English, with Hebrew subtitles. What Hebrew programming there is, my friend tells me, is of unusually high quality. Thus, watching TV in Israel usually means practicing reading and learning a foreign language simultaneously: a vastly richer activity, in terms of cognitive engagement, than watching TV in most countries. I’d like to do a randomized study in which you show students subtitled foreign-language films, say, twice a week. I wonder what the impact on educational outcomes would be.

Table 2: Summary of Correlations Between Performance on PISA Exams and Television Ownership

Organized in descending order by mean difference in score between students owning one television and those owning three or more. A cautionary note: standard errors on the performance of 1-TV students is quite high in many countries, especially the US, so we cannot be as confident in these numbers as I would like.

Country
GDP/Capita
Percentage of students who report owning three or more TVs
Performance of 1-TV kids
Performance of all students
Mean Difference in Score Between 1-TV Kids and 3-or-More-TV Kids
Mean Score
Rank in the OECD
Mean Score
Rank in the OECD
Luxembourg
82456.00
58.04
514.67
18th
481.67
29th
43.33
France
32495.00
49.63
525.33
14th
497.00
18th *
41.67
Australia
37615.00
67.2
553.33
1st
518.67
6th *
40.67
Belgium
34662.00
43.22
534.67
8th
509.33
11th
39.33
US
46434.00
78.86
529.67
11th
496.33
20th
38.33
Slovenia
26557.00
34.22
525.00
15th
498.67
17th
37.00
Iceland
36325.00
65.06
532.67
10th
501.00
12th *
35.67
Switzerland
41800.00
29.1
534.33
9th
517.33
8th
33.33
New Zealand
27020.00
58.29
543.67
3rd *
524.00
5th
29.33
Austria
36839.00
51.85
507.33
22nd
486.67
26th
28.00
Denmark
36326.00
78.75
526.33
13th
499.00
16th
27.33
Netherlands
39594.00
66.8
536.67
7th
518.67
6th *
20.67
Germany
34683.00
78.5
529.00
12th
510.00
10th
20.67
UK
34957.00
53.54
518.33
17th
500.00
15th
20.67
Japan
33635.00
49.28
543.67
3rd *
529.33
3rd
20.00
Canada
36397.00
69.27
542.33
5th
526.67
4th
19.67
Italy
31016.00
65.72
502.33
25th
486.00
27th
19.67
Sweden
36785.00
66.97
511.67
19th
495.33
22nd
19.00
Czech Rep.
23995.00
37.34
504.00
23rd
490.33
23rd
16.33
Finland
35322.00
58.96
553.00
2nd
543.67
1st
15.67
Norway
53672.00
51.6
511.33
20th
500.33
14th
15.67
Hungary
18763.00
46.96
510.00
21st
495.67
21st
13.33
Slovak Rep.
20270.00
28.55
497.00
26th
488.00
25th
11.33
Greece
27793.00
47.57
480.00
29th
473.00
30th
10.67
Korea
26574.00
7.92
542.00
6th
541.00
2nd
10.33
Estonia
20620.00
34.42
521.67
16th
513.67
9th
10.00
Spain
31469.00
56.78
490.67
28th
484.00
28th
7.33
Ireland
44381.00
77.86
497.00
26th
497.00
18th *
2.00
Poland
16312.00
29.39
502.67
24th
501.00
12th *
-2.33
Portugal
22638.00
70.47
478.00
30th
489.67
24th
-14.67
Chile
14106.00
52.53
426.00
33rd
439.00
33rd
-25.67
Israel
26444.00
50.3
440.00
31st
458.67
31st
-32.67
Mexico
14128.00
36.33
402.00
34th
420.00
34th
-38.00
Turkey
13362.00
10.08
437.33
32nd
454.33
32nd
-54.00
OECD Ave.
32218.97
51.8047059
508.93
 17th
496.62
17th
14.11

* Ranks marked with an asterix (*) are ties.
† Mean scores in this table are averages of scores in all three subject tests. Such an average is a bit of a Frankenstein Monster, but they make for a very concise table. Anyone who wants more detail can email me at deweytodelpit@gmail.com, and I’ll send you an excel spreadsheet with all of the data on television ownership and test scores.




[1] ^ PISA 2009 Results, Volume II, Table II.3.1

[2] ^ I estimated the math and science performance of students in the top ESC quartile by barbaric means. Lacking access to multi-level data, I simply looked at the PISA scores of students whose families owned three or more cars. Those students, who make up a stunning 46.43% of the population, placed 7th, 16th, and 10th in the OECD, in terms of reading, math, and science, respectively. Anyone interested in examining this question further should play around with the data querying tool on the PISA website. I warn you, though, it took a lot of Excel wrangling to turn those PHP tables into something that would reveal international rankings.

[3] ^ PISA 2009 Results, Volume II, Table II.3.1

[4] ^ Ibid

[5] ^ PISA 2009 Results, Volume II, Table II.5.10

[6] ^ Ibid

[7] ^ Ibid

[8] ^ PISA 2009 Results, Volume I, Table I.2.20

[9] ^ PISA 2009 Results, Volume II, Table II.3.1

[10] ^ PISA 2009 Results, Volume II, Table II.2.4

[11] ^ All of these data are drawn from the data-query tool on PISA’s website. If anyone wants to see all the data I pulled in spreadsheet form, just email me at deweytodelpit@gmail.com.

2 comments:

  1. In the section "Separate is not equal," you make a very interesting point when you state that "each student suffers not only his own disadvantages but the combined disadvantages of all his peers." I wonder if this same principle couldn't be applied on a larger scale to explain the disparity between the top ESC quartiles in the U.S. and Finland. In other words, large internal disparities within a country do not leave the seemingly insulated, economically-advantaged students unscathed... since, as we all know, no man is an island. For the same reason, I think indices that measure wealth inequality, such as the Gini-coefficient/Lorenz curve, are a more valuable approach to assessing the prosperity of a national economy than merely GDP.

    ReplyDelete
  2. Well, you make a good point, Anna, but I don’t think the effect you’re talking about can account for the US-Finnish score gap. PISA’s official publications make note of the effect of a country’s socioeconomic inequality on test scores. They estimate this effect by looking, not at the Gini index—which measures income inequality (using the Lorenz Curve)—, but at their own custo index of economic, social, and cultural status (ESCS), which they calculate based on parental job status and educational attainment, wealth-indicators like car- and dishwasher-ownership, availability of educational resources at home, and possession of cultural artifacts, such as works of art, poetry, and literature. This measure, as I’ve mentioned in my posts, is much more salient to educational outcomes than any simple variable like family income. Though the US has fairly high ESCS inequality, this is not as extreme as its income inequality.

    PISA also chooses to estimate the effect of inequality by looking, not at overall inequality in ESCS, but at the share of students who are disadvantaged—specifically, the share who are in the bottom 15.9% (one SD below the mean) of all students in the OECD by ESCS. Logically, I think this makes sense, because a country with a very high mean ESCS will have fewer disadvantaged students than another country with the same ESCS-inequality but a lower mean ESCS. With its medium-high ESCS inequality and medium-high mean ESCS, the US is pretty mid-range in terms of its share of disadvantaged students, but because several countries have extremely large shares of disadvantaged students, the average share for the OECD is actually larger than that of the US. That said, Finland’s share is quite low, and adjusting for share of disadvantaged students shrinks the US-Finnish reading performance gap by about 23%—from 36 points to 28 points. I mentioned this in my first Finland post, actually.

    ReplyDelete