As an example, for years, published research on state vocational rehabilitation programs reported big benefits — but only by comparing “apples” (people who successfully completed rehab and training) with “oranges” (participants in rehab who dropped out early or were not successfully rehabilitated). Subsequent gold standard studies, comparing apples and apples, did not bear out these results. As a result of awareness of the new, accurate findings, agencies running disability programs have moved from providing employment services to strengthening the financial incentives to work for this population.
3. Look for Random Assignment. Give findings from randomized controlled trials the greatest credence. Just as medical researchers run a lottery to give the new drug they are testing to some and a placebo to others, studies of social issues randomly divide a population of interest into a “treatment group” that receives an intervention and a “control group” that does not. These are “apples” and “apples” comparisons of families or individuals taken from the same tree.
4. Don’t Overinterpret Negative Findings. Scientists set a very high bar for proving a social program works, so it is possible that many programs that do work fail to meet this standard — often because of small study sample sizes. Researchers often fall into the trap of presuming studies that fail to prove program effectiveness have proved program failure. Don’t do this. “Statistically insignificant” findings have simply failed to show clear patterns in either direction and should not be the death knell for programs until clearer evidence of failure emerges. In some instances, such as the national Head Start study, the investigators, including myself, caution that “we cannot with this study sample make a confident conclusion either way.” This is a scientific truth lost in the public wrangling over where the study points in terms of policy.
Following these gold standard criteria, policymakers can make better-informed decisions about directing government dollars to expand the programs that are working and fine-tune or eliminate policies that are not working. The economic situation may push us to make quick decisions about our nation’s future, but we owe it to everyone to be sure we’re making them based on sound science.
Stephen Bell is a principal associate/scientist and a senior fellow with Abt Associates.