Dec. 20, 2014 SIGN IN | REGISTER
Download CQ Roll Call's Definitive Guide to the 114th Congress | Sign Up for Roll Call Newsletters | Get the Latest on the Roll Call App

Bell: Base Policy Decisions on the Best Research

As an example, for years, published research on state vocational rehabilitation programs reported big benefits but only by comparing apples (people who successfully completed rehab and training) with oranges (participants in rehab who dropped out early or were not successfully rehabilitated). Subsequent gold standard studies, comparing apples and apples, did not bear out these results. As a result of awareness of the new, accurate findings, agencies running disability programs have moved from providing employment services to strengthening the financial incentives to work for this population.

3. Look for Random Assignment. Give findings from randomized controlled trials the greatest credence. Just as medical researchers run a lottery to give the new drug they are testing to some and a placebo to others, studies of social issues randomly divide a population of interest into a treatment group that receives an intervention and a control group that does not. These are apples and apples comparisons of families or individuals taken from the same tree.

4. Dont Overinterpret Negative Findings. Scientists set a very high bar for proving a social program works, so it is possible that many programs that do work fail to meet this standard often because of small study sample sizes. Researchers often fall into the trap of presuming studies that fail to prove program effectiveness have proved program failure. Dont do this. Statistically insignificant findings have simply failed to show clear patterns in either direction and should not be the death knell for programs until clearer evidence of failure emerges. In some instances, such as the national Head Start study, the investigators, including myself, caution that we cannot with this study sample make a confident conclusion either way. This is a scientific truth lost in the public wrangling over where the study points in terms of policy.

Following these gold standard criteria, policymakers can make better-informed decisions about directing government dollars to expand the programs that are working and fine-tune or eliminate policies that are not working. The economic situation may push us to make quick decisions about our nations future, but we owe it to everyone to be sure were making them based on sound science.

Stephen Bell is a principal associate/scientist and a senior fellow with Abt Associates.

comments powered by Disqus

SIGN IN




OR

SUBSCRIBE

Want Roll Call on your doorstep?