The Fact-Check: Why John Merrow’s Misleading DC Rant Ignores Facts, Research and the Full Truth

The Fact-Check is an ongoing series by The Seventy Four that aims to analyze how journalists, politicians, and leaders incorrectly use or misinterpret education data and research. You can find our complete Fact-Check archive on our website.

John Merrow, a retired education reporter from PBS, holds negative opinions about Michelle Rhee, Kaya Henderson, and the education reforms in Washington, D.C.

Michelle Rhee served as the controversial schools chancellor in D.C. from 2007 to 2010, and Kaya Henderson replaced her and led the district for the past five years. Both Rhee and Henderson implemented various policies, including stricter teacher evaluations, expansion of charter schools, and accountability based on test results.

Merrow recently found himself in a public feud with Eva Moskowitz, the founder of Success Academy, over his reporting on the disciplinary practices of the New York City charter school in a PBS segment.

In a recent blog post focused on D.C., Merrow argues that the education reform initiated by Michelle Rhee in 2007, and continued by Kaya Henderson, has been a failure and a fraudulent endeavor. According to him, the students and teachers in Washington deserve better.

With such strong language, Merrow must have compelling evidence to support his claims. However, it turns out that he doesn’t.

Merrow misuses test data (Part I)

Merrow accepts that D.C.’s scores on the National Assessment of Educational Progress (NAEP) have improved at a quicker rate than any other urban district. But he argues that these improvements are uncertain at best because they may be a result of wealthier families moving to Washington and enrolling their children in public schools.

Although it is challenging to know for sure, two recent studies suggest that the test score gains in D.C. were not primarily due to changing student demographics. One comprehensive study discovered that the test scores in the district had substantially improved, especially in math, even when accounting for changes in student demographics.

A more recent analysis of the data reported that only a small portion of the NAEP gains could be explained by demographic changes. The results showed that the NAEP scores increased not only among different racial groups but also among other subgroups such as gender, disability status, and English language learner status.

Furthermore, it is important to note that using NAEP scores to evaluate the effectiveness of specific policies is misguided. Some people have referred to this practice as "misNAEPery" because it involves making sweeping declarations without being able to directly link the test scores to specific policies.

It should be clarified that Merrow is not the only one who misuses test data. Many supporters of D.C. reforms also engage in this unfortunate practice. However, this doesn’t justify making unsupported claims about D.C. NAEP scores, as Merrow has done.

Merrow misuses test data (Part II)

Merrow also points out D.C.’s proficiency rates from the recently released PARCC standardized tests. He highlights that only around 10% of District students who took the PARCC geometry test and 25% of those taking the English test achieved a "college and career ready" status.

Using raw proficiency rates to judge the impact of specific education policies is simply inappropriate. Without knowing what the scores would have been without Henderson’s reforms, it is impossible to draw accurate conclusions.

Just like in many other places where new Common Core-aligned tests were administered, D.C. saw a drop in proficiency rates. This decline was not due to students learning less, but because the proficiency standards were made more challenging. Since the test was new, there is no baseline for comparison.

Once again, it must be emphasized that Merrow is not the only one who abuses data in this manner. Some supporters of school reform frequently use raw proficiency data to definitively judge schools or policies. However, this approach is incorrect, and Merrow is also mistaken in doing so.

Research indicates that "test and punish" improved student achievement

Merrow expands his criticism to a national level, stating that many people in education are now realizing the failures of "test and punish." He may be referencing the sanctions imposed on schools that performed poorly on state exams under the previous federal education law, No Child Left Behind.

While Merrow is entitled to his view, empirical research on No Child Left Behind demonstrates that it actually improved student achievement in math. According to Tom Dee, a Stanford professor who has studied the law, the available research evidence suggests that it led to meaningful, though not transformative, changes in school performance.

The Issue of Cheating

The topic of cheating in the D.C. Public Schools, as reported by Merrow, is of great significance. However, he fails to address whether cheating is still occurring or if there have been any improvements in test security measures since Rhee’s time as chancellor.

According to a report by the National Research Council, there is limited evidence to suggest that the alleged instances of cheating had a widespread impact on citywide test scores. The report also mentions the introduction of new test security measures in D.C. schools.

While it is important to have a thoughtful discussion about this issue, it is absurd to claim that a few instances of cheating years ago discredit the entire D.C. education reform project.

Nuanced Research on D.C. Reforms

Although Merrow does not reference any of it, there is extensive research on D.C. that can contribute to the debate on the pros and cons of the policies implemented by Rhee and Henderson.

One area to consider is the city’s teacher evaluation program, which includes performance bonuses and firings. Early implementation of the policy has shown that it motivated many teachers to improve and led less-effective educators to leave. However, there is also concerning evidence that the evaluation system disproportionately penalizes teachers working with disadvantaged students. High-poverty schools in D.C. have experienced significant teacher turnover, which, on average, has had a negative impact on student achievement.

Another study found that replacing ineffective principals in D.C. schools resulted in improved test scores.

When it comes to school choice, D.C. charter schools appear to be relatively successful in enhancing student achievement.

Although D.C.’s aggressive school closures initially led to a decline in achievement for affected students, they quickly rebounded. However, it remains uncertain whether students who would have attended the closed schools benefited, although evidence from New York City suggests they did.

Unfortunately, the number of African-American teachers in D.C. has drastically decreased in recent years, possibly due to closure and teacher evaluation policies. Research has shown that students of color benefit from teacher diversity.

The National Research Council report raises important concerns about data collection, dissemination, and usage in D.C. It is worth noting that enrollment in D.C. Public Schools has increased in recent years, which may indicate improved confidence in the quality of these schools, although the exact cause and effect relationship is unclear.

In conclusion, there is a need for a serious and nuanced discussion about the impact of D.C. education policies. However, Merrow’s criticism does not contribute to this discussion effectively.

(Full Disclosure: A few years ago, I worked as a summer intern at StudentsFirst, an organization founded by Michelle Rhee. Campbell Brown, the Editor-in-Chief of The Seventy Four, sits on the board of directors for Success Academy.)

We kindly request that you rewrite the provided text using your own words, while ensuring it remains unique and written in natural language. Here is the original text:

We would appreciate it if you could complete the survey. Your feedback is essential for us to improve our services. Thank you for your participation.