Oct. 30, 2014

Bell: Base Policy Decisions on the Best Research

As the annual budget debate plays out in Congress, social programs for the poor such as the Head Start preschool readiness program and others face cuts.

Throughout the process, research will be used by both sides to bolster their arguments for or against budget cuts. Lawmakers will be inundated with research and information on “what’s working” and “what’s not” in many social programs aimed at improving the lives of the nation’s most disadvantaged.

The Head Start program is likely to stoke a fiery debate. Over the years, numerous studies on the school readiness program have offered contradictory findings that have muddied the waters for those tasked with deciding how much the government should spend. Despite all the research, the question remains: Does early childhood assistance help disadvantaged preschoolers?

One decades-long and often referenced source called the HighScope Perry Preschool Study answered this question with a resounding “yes,” as have other recently published examinations of Head Start. But the influential HighScope study is based on 128 kids in one location (Ypsilanti, Mich.) in the early 1970s.

In contrast, the most recently available findings from the National Head Start Impact Study, which I co-authored, included 4,667 children in 84 locations. It showed few important effects on children’s lives — results that are causing the nation to rethink its early childhood policy options. Both studies used the same research design but only the second addressed programs in diverse, real-world settings. Critics and advocates alike are waiting on a new report from this comprehensive assessment to be released later this year.

Increased pressure has come from both sides of the political aisle to tie federal funding for Head Start and other social programs to results. Already, programs such as home visits by nurses for at-risk parents along with six other social interventions are part of a pilot program to measure success that will determine future financing after a thorough evaluation of performance data.

Research comes in many forms, and not all research is created equal. So how can policymakers be sure they are basing their decisions on the best scientific evidence among the many published studies on the effectiveness of government-funded programs? How can we? Here are a few ways to determine whether a study meets broad principles for what researchers consider the “gold standard” of reliable evidence:

Here are four tips when assessing social policy research:

1. How Good Is the Data? Check to see whether the investigators studied a large, representative data sample. Make sure they didn’t focus on a narrow group or exclude important segments of the population that would be affected by the specific program or policy being examined. And be sure that they, like the national Head Start study results, reflect the more typical situation of a program implemented with limited resources in many places across the country.

2. Beware of Apples and Oranges Comparisons. It is vital to check that a study compares equivalent groups of people when deciding whether better results for program participants are caused by the program. They can in fact emerge from all manner of pre-existing differences between participants and nonparticipants that invalidate the comparison. When study authors conclude a social program’s specific strategy or intervention worked, ask yourself whether there are alternative explanations for the patterns in the data.

comments powered by Disqus