This study presents a lengthy but incomplete and misleading analysis of the implementation and impact of Chicago's high stakes summer school program (Summer Bridge).

In Chicago's Summer Bridge program, students who have failed to meet an Iowa Test score cut-off in either reading and/or math in grades three, six, and eight in the spring have been required to attend a summer school program since summer 1997.  The Summer Bridge Program has focused almost entirely on preparing students to pass the Iowa Test cutoff scores.  Students who fail to meet the test score cut-off scores at the end of the summer in the subjects or subjects that they failed in the spring have been required to repeat a grade (retained or flunked).  Students who meet the required cutoff scores at the end of the summer have been promoted.  (Some students who fail the test have still been promoted through a "waiver" process.)

This Comment makes two major points about the shortcomings of the Consortium's study:

The Long-Term Impact of Summer Bridge on Retained Students is Ignored

While billed as a study of "The Results from Summer Bridge," this study does not analyze the long-term effects of the summer program on those Summer Bridge students who were retained.  In 1997 and 1998, for example, a total of 24,000 students were retained at the end of Summer Bridge. [4]  Once retained, only about 50% of retained students ever met the cutoff score for promotion, even after repeating a year of school and often after a second Summer Bridge experience.  Depending on the grade, the Consortium's own research showed that retained students were performing worse than or no better than students who had been socially promoted before reform and had received no special help and no extra year of school. [5]   Yet the cost of Summer Bridge, plus the cost of an extra year of schooling for retained students, are about $100 million per year.

Elsewhere, Melissa Roderick, the senior author of the current study has bluntly acknowledged the enormous harm of Chicago's retention process on the retained students:

Yet the long-term costs and benefits of Summer Bridge for retained children are not analyzed in a report titled "Results from Summer Bridge."

Another negative impact of Summer Bridge for retained students is obscured in the report.  Prominently featured as a "Key Finding" on page 2 of the report is the following statement:

However, one must read the report with extreme care (such as scrutinizing charts on page 46) to find out that about half of the "high risk" students in Summer Bridge were retained.  And you must read page 111 of the report to find a key question raised about the actual benefits of Summer Bridge even for those "high risk" students who made big summer gains:

Is it really "one of the most positive findings" of Roderick's report that "Third graders who were at the highest risk of failure benefited most from the program," when the net effect of their participation was that 50% of these students were retained in a process that Roderick describes elsewhere as "very decimating" and "just a disaster" for the retained students?

No Educationally Significant Long-Term Benefit, Even for Promoted Students

As Harris Cooper, an expert in analyzing the impact of summer school states, "The investment of resources in summer programs is wasted if it does not result in lasting benefits for participating students." [9]   (emphasis added).  Chapter 6 of the Summer Bridge report (pages 99-107) addresses this bottom-line question of long-term impact.  The results show that the program does not result in any educationally significant long-term gains, even for those students who were promoted at the end of the summer.

The researchers compared the long-term achievement of low-scoring students who just barely failed to meet the spring test score cut-offs and thus had to attend Summer Bridge (Summer Bridge High Scorers), to low-scoring students who just barely met the cut-offs in the spring and did not have to attend (Comparison Group).  As the authors state, "In many cases, this decision [about whether the students went to Summer Bridge or not] resulted from answering one question right or wrong on the reading test." [10]   The researchers made four comparisons between the longer-term achievement gains of Summer Bridge Higher Scorers and the relevant Comparison Groups (at third and sixth grades for students retained in summer 1997 and summer 1998).

The researchers argue that Summer Bridge "appears to narrow the gap" between two low-achieving groups of students (the Summer Bridge High Scorers who were promoted and the Comparison Group), but does not substantially improve the long-term Iowa Test performance of students who attended Summer Bridge." [11]

Even this very limited conclusion about the benefits of Summer Bridge for the promoted students is not justified by the data.

There are three reasons why this analysis did not show an "educationally significant" improvement as a result of Summer Bridge.

First, while the researchers analyzed the students results in "logits," it appears that the two groups of students typically differed, on average, by about two months in grade equivalents (or GEs)  in the spring before Summer Bridge.  Yet the four comparison charts on page 104 of the study show that in no instance did the "Summer Bridge High Scorers ever score higher than their Comparison Group." (See relevant tables from the Consortium study).  The authors use sophisticated statistical analysis to show that there was a slight narrowing of the gap between these two low-achieving groups of students, which in most instances appears to be about one month at best.  Both groups remained very low-scoring.  Is this slight narrowing worth the $13 million annual investment in Summer Bridge, when other approaches (such as strengthening the regular school year program) are so much more promising?  And is this tiny benefit worth it when the student retention resulting from Summer Bridge does so much harm?

Second, the small long-term benefits of Summer Bridge rest entirely on gains on the Iowa Test, the test for which Chicago students have been intensively prepped during the regular year and the summer since 1977.  Examining Chicago's high stakes testing program, the Chair of a committee of the National Research Council that analyzed high stakes testing concluded that:

It is doubtful that the tiny benefits documented in the study would hold up if students were given a test with a slightly different format that covered the same objectives (but for which they had not been prepped), a point that has been established by previous research about testing. [13]

Third, Note 4 on page 100 of the report (which is explained on page 135) indicates that about 20% to 30% of the Higher Scorer Summer Bridge groups were eliminated from the four comparisons made on page 104 (See relevant tables from the Consortium study).  Specifically, these students were eliminated because they flunked after Summer Bridge, even though they had entered the program with scores very close to the passing score.  It is likely that the students who were dropped from the analysis were less motivated or less able than the Summer Bridge High Scorers who were promoted (and included in the analysis).  Given the tiny differences between the Summer Bridge High Scorers and the Comparison Groups, the decision to drop 20% to 30% of the least able students from the Summer Bridge High Scorers makes the study's conclusion about a small benefit for the Summer Bridge High Scorers even more questionable. [14]

As Designs for Change has analyzed in detail in other publications, there are research-based alternatives that should be carried out that are superior to both retention and social promotion. [15]   The latest Consortium study, combined with other existing evidence, shows that few children have benefited long-term from Chicago's massive flunking program and tens of thousands of children have been harmed.

NOTES