Study of Between-States Comparisons
Methodological Study of Between-States Comparisons, With Particular Application To .08% BAC Law Evaluation
Author: Robert A. Scopatz, Ph.D. Data Nexus, Inc. 1040 Jason Ridge Court Kissimmee, FL 24747-1238 407/397-0574 firstname.lastname@example.org
Transportaion Research Board
77th Annual Meeting January 11-15, 1998
For a complete survey, contact the NMA at 608/849-6000 or 402 W. Second Street, Waunakee, WI 53597 or email at email@example.com
Hingson, Heeren and Winter (1996) published results purporting to show that lowering legal Blood Alcohol (BAC) limits to .08% resulted in a 16% reduction in the probability that a fatally injured driver would have a BAC above that level. Their results also show that broadening the picture to include all drivers in fatal crashes (for whom BAC content is known) yields the same result. Hingson et al. conclude that passing a law caused a sustained decrease in the chance that a driver involved in a fatal crash has a blood alcohol level exceeding the .08% limit. *The authors estimate that if all states passed .08% BAC laws, between 500 and 600 lives would be saved annually.
There are, however, numerous differences between the “law” and “comparison” states in the Hingson et al. study in addition to the passage of a law in the “law” states. Since these differences exist, it is impossible to conclude that the passage of a law, as opposed to some other, uncontrolled-for factor, accounts for the results. It is important, from a methodological standpoint, to test the reliability of the results before concluding that the paired comparisons between states has demonstrated a true effect of .08% BAC laws.
The following is a replication of the Hingson et al. (1996) study using first, the same state pairs as were used in the original, and second with a substitute set of comparison states drawn from among those states which have retained the .10% legal BAC limit at least to the end of the study period (data are available from FARS up to 1994). A third replication was tested in which only the state paired with California was changed. A further replication also changed the comparison to California, but this time used a group of states which together closely matched California in land area, population, and number of fatal crashes.
NMA NOTE: Although the analysis of the Hingston et al. study discredits this assumption, it fails to mention that .08% BAC laws do not appear to reduce the total of fatal accidents, rather there is a shift in the ration of “alcohol-related” accidents to non-alcohol-related accidents. The latter increases in proportion to the former’s decline. This strategy suggests that alcohol per se was not a consistent factor in this group of fatal accidents.
The Fatal Accident Reporting System (FARS) data released on CD-ROM were used throughout this study. Using the same methodology as presented in the original Hingson et al. study, a set of five state pairs were selected for analysis of the change in proportion of drivers in fatal crashes with a BAC above .08%. The first member of each state pair was a state that had implemented a legal BAC limit of 0.08%. The second member of each state pair was a “comparison” state. In the original study, Hingson et al. selected the comparison states based on their geographic proximity to the “law” states, and on their perceived similarity to those states.
The current study was designed to perform four replications of the Hingson et al. study. In the first replication, the same state pairs as those used by Hinson et al. were used. This represented an attempt to exactly replicate the original study in order to validate that the same results would be found using the more recent release of FARS data. The second replication used a different set of comparison states paired with the original “law” states. The third replication changed only the comparison state paired with California. The fourth replication changed the comparison to California, but this time used a combination of three states which, together, closely matched California’s population, land area, and the total number of fatal crashes. In all replications, the same “law” states and the same pre-/post-time durations were used as in the original study. As in the original study, the time durations varied for the different state pairs based on the year and month in which the law was enacted in the “law” state, and the number of years of FARS data available.
Across four replications of Hingson, Heeren, and Winter (1996), the analyses presented fail to support those authors’ conclusions. The reason for this is that the paired comparison methodology used in the Hingson, et al. study is extremely sensitive to changes in the comparison states chosen for analysis. If logically valid, but different comparison states are chosen, the results change dramatically. In fact, in every case, the overall “ratio of change” is close to 1.00 – the ratio one would expect if there were no statistically significant effect of .08% BAC laws.
The following conclusions are warranted by this analysis:
- The between-state comparison methodology should be used with extreme caution. If, as in the four replications here, the results are sensitive to the particular states chosen, the only conclusion to be drawn is that the results are not robust; i.e., they should not be generalized.
- Making the causal inference, as was done in the original study, that .08% BAC laws had an affect on the proportion of fatally injured drivers showing an illegal BAC, is not supported. It is generally not considered valid to draw causal inferences from the kind of quasi-experimental design used in this study, primarily because you cannot control all confounding variables. As shown in the four replications, there is ample evidence to believe that extraneous variables were responsible for the results produced in the original Hingson, et al. (1996)study.
- If between states comparisons are to be used, it is recommended that the researchers attempt to replicate their findings under varying conditions to be sure that they are robust and repeatable. The tests conducted here involved changing the comparison states in some or all of the state pairs. Had the original findings been replicated, the result might still be said to be caused by some confounding factors, however, the convergent evidence across several replications would at least tend to support the conclusion that the laws were a primary contributing factor to the declines reported. In this case, the replication effort failed to provide that converging evidence.
Hingson, R., Heeren, T., and Winter, M. Lowering State Legal Blood Alcohol Limits to 0.08%: The Effect on Fatal Motor Vehicle Crashes. American Journal of Public Health, Vol. 86, No. 9, pp., 1297-1299.
Bureau of Transportation Statistics. Traffic Safety CD-ROM: Fatal Accident Reporting System (FARS): 1975-1994 and General Estimates System (GES): 1988-1994, 1996, United States Department of Transportation.
National Highway Traffic Safety Administration, National Center for Statistics and Analysis. Traffic Safety Facts 1994: A compilation of Motor Vehicle Crash Data from the Fatal Accident Reporting System and the General Estimates System. August 1995, United States Department of Transportation.