Call Us: US - +1 845 478 5244 | UK - +44 20 7193 7850 | AUS - +61 2 8005 4826

Evaluating a Change in Outcomes, and Impacts

Unintended Consequences Consider potential unintended consequences that may occur as a result of the policy implementation. Rely on previous research and evaluations and the experience of stakeholders to brainstorm potential unintended consequences. Some unintended consequences may be uncovered during the course of policy implementation. Some examples of potential unintended consequences include: ƒ Increases in the arrest of intimate partner violence victims as a result of a new arrest policy. ƒ Increases in illegal firearm sales as a result of a firearm licensing policy. ƒ Increases in child injuries due to airbag deployment as a result of new regulatory requirements. ƒ Issues related to access to health care as the result of policies that increase reporting of injuries. Estimating the Cost-Benefit of a Policy To examine the cost savings associated with adopting the Brain Trauma Foundation (BTF) guidelines for treatment of severe traumatic brain injury, Faul and colleagues used surveillance systems combined with national surveys. They estimated the lifetime costs of 80% adherence to the guidelines compared with the 33% estimated adherence. Using a decision analysis model, coupled with previous research and available surveillance and survey data, they estimated savings of more than $300 million in medical costs and rehabilitation costs if the BTF guidelines were followed at 80% adherence. Faul’s team also estimated that more than 3,000 additional lives would be saved. This example demonstrates how previous research and available surveillance data can estimate the cost benefits of a policy.8 5 Step by Step – Evaluating Violence and Injury Prevention Policies Potential Policy Impact Evaluation Challenges and Solutions Challenges Solutions External and contextual factors such as economic conditions or public awareness ƒ Measure contextual factors to the extent possible. ƒ Explore the use of difference in difference analyses which examine the difference in the target group while accounting for differences in comparison communities. Length of time required to expect long-term impacts ƒ Use an evaluation plan that measures short-term and intermediate outcomes that logically link to long-term outcomes. Lack of access to appropriate data ƒ Identify available pre-existing datasets and explore the possibility of data linkage to increase analysis possibilities (see Brief 6). Action Steps ƒ Identify any resources for planning and implementing an impact evaluation. ƒ Identify evaluation questions and identify the most appropriate design given available resources and expertise. ƒ Articulate short-term and intermediate outcomes as well as long-term impacts for a particular policy. ƒ Identify data collected in an existing surveillance or administrative system to use for an evaluation. Additional Resources The Magenta Book: Guidance for Evaluation (Her Majesty’s Treasury). Provides general and technical guidance on policy evaluation. Available at Policy Evaluation Webinar Series (National Collaborative on Childhood Obesity Research). Available at http://