Thomas F. Osborne

ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We show that this problem appears in a wide variety of practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

Base editors can correct disease-causing genetic variants. After a neonate had received a diagnosis of severe carbamoyl-phosphate synthetase 1 deficiency, a disease with an estimated 50% mortality in early infancy, we immediately began to develop a customized lipid nanoparticle-delivered base-editing therapy. After regulatory approval had been obtained for the therapy, the patient received two infusions at approximately 7 and 8 months of age. In the 7 weeks after the initial infusion, the patient was able to receive an increased amount of dietary protein and a reduced dose of a nitrogen-scavenger medication to half the starting dose, without unacceptable adverse events and despite viral illnesses. No serious adverse events occurred. Longer follow-up is warranted to assess safety and efficacy. (Funded by the National Institutes of Health and others.).

There is growing evidence that thrombotic and inflammatory pathways contribute to the severity of COVID-19. Common medications such as aspirin, that mitigate these pathways, may decrease COVID-19 mortality. This retrospective assessment was designed to quantify the correlation between pre-diagnosis aspirin and mortality for COVID-19 positive patients in our care. Data from the Veterans Health Administration national electronic health record database was utilized for the evaluation. Veterans from across the country with a first positive COVID-19 polymerase chain reaction lab result were included in the evaluation which comprised 35,370 patients from March 2, 2020 to September 13, 2020 for the 14-day mortality cohort and 32,836 patients from March 2, 2020 to August 28, 2020 for the 30-day mortality cohort. Patients were matched via propensity scores and the odds of mortality were then compared. Among COVID-19 positive Veterans, preexisting aspirin prescription was associated with a statistically and clinically significant decrease in overall mortality at 14-days (OR 0.38, 95% CI 0.32-0.46) and at 30-days (OR 0.38, 95% CI 0.33-0.45), cutting the odds of mortality by more than half. Findings demonstrated that pre-diagnosis aspirin prescription was strongly associated with decreased mortality rates for Veterans diagnosed with COVID-19. Prospective evaluation is required to more completely assess this correlation and its implications for patient care.

Background: Information about the effectiveness of oral antivirals in preventing short- and long-term COVID-19–related outcomes in the setting of Omicron variant transmission and COVID-19 vaccination is limited. Objective: To measure the effectiveness of nirmatrelvir–ritonavir and molnupiravir for outpatient treatment of COVID-19. Design: Three retrospective target trial emulation studies comparing matched cohorts of nirmatrelvir–ritonavir versus no treatment, molnupiravir versus no treatment, and nirmatrelvir–ritonavir versus molnupiravir. Setting: Veterans Health Administration (VHA). Participants: Nonhospitalized veterans in VHA care who were at risk for severe COVID-19 and tested positive for SARS-CoV-2 during January through July 2022. Intervention: Nirmatrelvir–ritonavir or molnupiravir pharmacotherapy. Measurements: Incidence of any hospitalization or all-cause mortality at 30 days and from 31 to 180 days. Results: Eighty-seven percent of participants were male; the median age was 66 years, and 18% were unvaccinated. Compared with matched untreated control participants, those treated with nirmatrelvir–ritonavir (n = 9607) had lower 30-day risk for hospitalization (22.07 vs. 30.32 per 1000 participants; risk difference [RD], −8.25 [95% CI, −12.27 to −4.23] per 1000 participants) and death (1.25 vs. 5.47 per 1000 participants; RD, −4.22 [CI, −5.45 to −3.00] per 1000 participants). Among persons alive at day 31, reductions were seen in 31- to 180-day incidence of death (hazard ratio, 0.66 [CI, 0.49 to 0.89]) but not hospitalization (subhazard ratio, 0.90 [CI, 0.79 to 1.02]). Molnupiravir-treated participants (n = 3504) had lower 30-day and 31- to 180-day risks for death (3.14 vs. 13.56 per 1000 participants at 30 days; RD, −10.42 [CI, −13.49 to −7.35] per 1000 participants; hazard ratio at 31 to 180 days, 0.67 [CI, 0.48 to 0.95]) but not hospitalization. A difference in 30-day or 31- to 180-day risk for hospitalization or death was not observed between matched nirmatrelvir- or molnupiravir-treated participants. Limitation: The date of COVID-19 symptom onset for most veterans was unknown. Conclusion: Nirmatrelvir–ritonavir was effective in reducing 30-day hospitalization and death. Molnupiravir was associated with a benefit for 30-day mortality but not hospitalization. Further reductions in mortality from 31 to 180 days were observed with both antivirals. Primary Funding Source: U.S. Department of Veterans Affairs.