Evaluating efficiency of English acute foundation trusts under system reform: a two-stage DEA approach Abstract The English healthcare sector underwent extensive system reform over the period from 2010 to 2015, aimed principally at improving technical efficiency. This paper examines the effect of the reforms on foundation trusts in England with particular emphasis on technical efficiency. By employing data envelopment analysis (DEA) and a second-stage regression, we found evidence of an overall improvement in efficiency, notwithstanding some fluctuations. Specifically, we found that bed utilization had positive and statistically significant association with the efficiency of acute foundation trusts; suggesting that better management of patient flows and bed utilization might be expected to improve hospital efficiency. We also found evidence to suggest that efficiency might also be improved through better management of staff numbers, optimizing liquidity, and better utilization of assets such as buildings and information technology.

Assessing the impacts of governance reforms on health services delivery: a quasi-experimental, multi-method, and participatory approach Abstract Despite considerable advances in developing new and more sophisticated impact evaluation methodologies and toolkits, policy research continues to suffer from persistent challenges in achieving the evaluation trifecta: identifying effects, isolating mechanisms, and influencing policy. For example, evaluation studies are routinely hampered by problems of establishing valid counterfactuals due to endogeneity and selection effects with respect to policy reform. Additionally, robust evaluation studies often must contend with heterogeneity in treatment, staggered timing, and variation in uptake. And finally, on practical grounds, researchers frequently struggle to involve policymakers and practitioners throughout the research process in order to engender the type of trust needed for policy influence. While it can be difficult to generalize about appropriate evaluation methodologies across contexts, prominent policy interventions like governance reforms for improving health services delivery nonetheless demand rigorous and comprehensive evaluation strategies that can produce valid results and engage policymakers. Drawing on illustrations from our research on health sector decentralization in Honduras, in this paper we present a quasi-experimental, multi-method, and participatory approach that addresses these persistent challenges to policy evaluation.

Assessing the performance of direct and indirect utility eliciting methods in patients with colorectal cancer: EQ-5D-5L versus C-TTO Abstract Direct and indirect preference-based methods are two main types of utility generating methods which are being frequently used in cost-utility analysis. However, a key question is how do these two methods perform in terms of validity? The aim of this study is to assess the utilities derived from EuroQol five-dimensional 5 level (EQ-5D-5L) as an indirect method and composite time trade-off (C-TTO) as a direct method in patients with colorectal cancer (CRC). A total of 223 patients with CRC were consecutively selected from three cancer centers in Tehran between May and September 2017. The EQ-5D-5L and C-TTO methods were used to estimate utilities in patients. Wilcoxon test was performed to compare utilities between C-TTO and EQ-5D-5L. In addition, intraclass correlation coefficient (ICC) and Bland–Altman plot were used to assess the agreement between the two methods. Bootstrapping quantile regression analysis was performed for comparison coefficient. The mean values of EQ-5D-5L (0.7186 ± 0.1275) for all stages were significantly higher than the C-TTO (0.4180 ± 0.4670) values. Stage 1 had the highest utility in both C-TTO (0.8167 ± 0.2332) and EQ-5D-5L (0.8339 ± 0.0535), whereas stage 4 had the lowest utility (0.1283 ± 0.5530 and 0.6001 ± 0.1220, respectively). As well, the C-TTO underestimates the values in low level of utilities and overestimates in high level of utilities. The overall ICC between the two methods was 0.50. Our study provides some tentative results about the differences between values of EQ-5D-5L and C-TTO, but clearly more work is needed to be done on this issue.

Evaluating quality of hospital care using time-to-event endpoints based on patient follow-up data Abstract Revisions of hip and knee arthroplasty implants and cardiac pacemakers pose a large medical and economic burden for society. Consequently, the identification of health care providers with potential for quality improvements regarding the reduction of revision rates is a central aim of quality assurance in any healthcare system. Even though the time span between initial and possible subsequent operations is a classical time-to-event endpoint, hospital-specific quality indicators are in practice often measured as revisions within a fixed follow-up period and subsequently analyzed by traditional methods like proportions or logistic regression. Methods from survival analysis, in contrast, allow the inclusion of all observations, i.e. also those with early censoring or events, and make thus more efficient and more timely use of the available data than traditional methods. This may be obvious to a statistician but in an applied context with historic traditions, the introduction of more complicated methods needs a clear presentation of their added value. We demonstrate how standard survival methods like the Kaplan–Meier estimator and a multiplicative hazards model outperform traditional methods with regard to the identification of performance outliers. Following that, we use the proposed methods to analyze 640,000 hip and knee replacement operations with about 13,000 revisions between 2015 and 2016 in more than 1200 German hospitals in the annual evaluation of quality of care. Based on the results, performance outliers are identified which are to be further investigated qualitatively with regard to their provided quality of care and possible necessary measures for improvement. Survival analysis is a sound statistical framework for analyzing data in the context of quality assurance and survival methods outperform the statistical methods that are traditionally used in this area.

Accounting for study participants who are ineligible for linkage: a multiple imputation approach to analyzing the linked National Health and Nutrition Examination Survey and Centers for Medicare and Medicaid Services’ Medicaid data Abstract Data from the National Health and Nutrition Examination Survey have been linked to the Center for Medicare and Medicaid Services’ Medicaid Enrollment and Claims Files for the survey years 1999–2004. The linked data are produced by the National Center for Health Statistics’ (NCHS) Data Linkage Program and are available in the NCHS Research Data Center. This project compares the usefulness of multiple imputation to account for data linkage ineligibility and other survey nonresponse with currently recommended weight adjustment procedures. Estimated differences in environmental smoke exposure across Medicaid/Children’s Health Insurance Program (CHIP) enrollment status among children ages 3–15 years are examined as a motivating example. Comparisons are drawn across the three different estimates: one that uses MI to impute the administrative Medicaid/CHIP status of those who are ineligible for linkage, a second that uses the linked data restricted to linkage eligible participants with a basic weight adjustment, and a third that uses self-reported Medicaid/CHIP status from the survey data. The results indicate that estimates from the multiple imputation analysis were comparable to those found when using weight adjustment procedures and had the added benefit of incorporating all survey participants (linkage eligible and linkage ineligible) into the analysis. We conclude that both multiple imputation and weight adjustment procedures can effectively account for survey participants who are ineligible for linkage.

Modeling determinants of time-to-circumcision of girls: a comparison of various parametric shared frailty models Abstract Female genital mutilation (FGM), also known as female genital cutting or female circumcision, is one of the deeply rooted traditional practices, in which the external female genital organ is either partially or totally removed for non-medical reasons. In Ethiopia, FGM is widespread across the majority of regions and ethnic groups, having the highest national prevalence that leads them to various complications such as immediate urinary and genital tract infection, pain and hemorrhage, complications in childbirth and social, psychological and sexual complications. This study aimed to model and investigate the potential risk factors of time-to-circumcision of girls in Ethiopia using parametric shared frailty models where regional states of the girls were used as a clustering effect in the models. The data source for the analysis was the 2016 EDHS data collected from January 18, 2016 up to June 27, 2016 from which the survival information of 2930 girls on age at circumcision obtained. The gamma and inverse Gaussian shared frailty distributions with Exponential, Weibull and log-logistic baseline models was employed to analyze risk factors associated with age at circumcision using socio-economic and demographic factors. All the fitted models were compared by using AIC and BIC values from simulation study and actual dataset. The result revealed that about 22.4% of girls were circumcised and 77.6% were not circumcised. The median age at circumcision was 3 years. Based on AIC and BIC values from simulation experiment and graphical evidences, log-logistic model with inverse Gaussian shared frailty distribution preferred when compared with other models for age at circumcision dataset. The clustering effect was significant for modeling the determinants of time-to-circumcision of girls dataset. Based on the result of log-logistic inverse Gaussian shared frailty model, mothers and fathers educational level, place of residence and religion of parents were found to be the most significant determinants of age at circumcision of girls. The estimated acceleration factor for the group of mothers who had secondary and higher educational level were highly prolonged age at circumcision of girls by the factor of ϕ = 3.119 and ϕ = 3.933 respectively. The log-logistic model with inverse Gaussian shared frailty distribution described age at circumcision of girls better than other models and there was heterogeneity between the regions on age at circumcision. Improving parents access to education would be an important way approach for preventing girls’ circumcision.

Difference-in-differences and matching on outcomes: a tale of two unobservables Abstract Difference-in-differences combined with matching on pre-treatment outcomes is a popular method for addressing non-parallel trends between a treatment and control group. However, previous simulations suggest that this approach does not always eliminate or reduce bias, and it is not clear when and why. Using Medicaid claims data from Oregon, we systematically vary the distribution of two key unobservables—fixed effects and the random error term—to examine how they affect bias of matching on pre-treatment outcomes levels or trends combined with difference-in-differences. We find that in most scenarios, bias increases with the standard deviation of the error term because a higher standard deviation makes short-term fluctuations in outcomes more likely, and matching cannot easily distinguish between these short-term fluctuations and more structural outcome trends. The fixed effect distribution may also create bias, but only when matching on pre-treatment outcome levels. A parallel-trend test on the matched sample does not reliably distinguish between successful and unsuccessful matching. Researchers using matching on pre-treatment outcomes to adjust for non-parallel trends should report estimates from both unadjusted and propensity-score matching adjusted difference-in-differences, compare results for matching on outcome levels and trends and examine outcome changes around intervention begin to assess remaining bias.

Causal inference for multi-level treatments with machine-learned propensity scores Abstract Propensity score-based methods have been widely developed to adjust for confounders in observational studies to estimate causal treatment effect for binary treatments. We generalize these causal inference methods to the multi-level treatment case. We review the generalized causal inference framework and several propensity score estimation methods. We conduct a comprehensive simulation study to evaluate the performance of multinomial logistic regression, generalized boosted models, random forest and data adaptive matching score for estimating propensity scores based on inverse probability of treatment weighting. From our findings, multinomial logistic regression is susceptible to yielding extreme weights while a mis-specified model is assumed, which results in poor performance of the inverse probability weighted estimator. On the other hand, machine-learned propensity scores tend to have less biased and more stable performance, and the data adaptive matching score tends to perform the best overall. The above-mentioned propensity score based methods are applied to the Taobao dataset to evaluate the causal effect of reputation on sales.

Developing and evaluating methods to impute race/ethnicity in an incomplete dataset Abstract The availability of race data is essential for identifying and addressing racial/ethnic disparities in the health care system; however, patient self-reported racial/ethnic information is often missing. Indirect methods for estimating race have been developed, but they usually only consider geocoded and surname data as predictors, may perform poorly among racial minorities, they do not adjust for possible errors for specific datasets, and are unable to provide race estimates for subjects missing some of this information. The objective of this study was to address these limitations by developing novel methods for imputing race/ethnicity when this information is partially missing. By viewing the unobserved race as missing data, we explored different multiple imputation methods for imputing race/ethnicity, and we applied these methods to a subset of Rhode Island Medicaid beneficiaries. Current race imputation methods and newly developed ones were compared using area under the ROC curve statistics and racial composition estimates to identify methods and sets of predictors that yield superior race imputations. Family race was identified as an important predictor and should be included in race estimation models when possible. Bayesian regression models (BRM) provide better race estimates than previously proposed methods. Missing race was multiply imputed using joint modeling and fully conditional specification. Post-imputation analyses showed that fully conditional specification with a BRM is superior to joint modeling for race imputation. The proposed fully conditional specification method is a flexible, effective way of estimating race/ethnicity that allows for propagation of imputation error and ease of interpretation in further analyses.

A conversation with Sally C. Morton: excellence in health policy statistics Abstract Sally C. Morton is internationally recognized in the use of statistics in health policy, and has had a career with incredible impact as evidenced by her many leadership roles. She was awarded the ASA Health Policy Statistics Section’s (HPSS) Long-Term Excellence Award in January 2018 at the 12th International Conference on Health Policy Statistics. Morton is currently the Dean of the College of Science at Virginia Tech. This article is conversation with Morton about her career.