Better in their words…

Introduction

When Vinay Prasad and I first published Ending Medical Reversal, not that many people read it.¹ Opinions of those who did tended to track with the readers’ faith in medicine. People who were already a bit skeptical of medical practice said the book confirmed their belief that, although we preach evidence-based medicine, we practice something else. Those who were less questioning about 21^st-century medicine either thought we were wrong or told us that it made them a bit more skeptical. People who had no faith in conventional medicine celebrated the book, but if they were alternative medicine devotees, they tended not to extend the intended message to that discipline.

I have thought of this third group more and more recently. We’ve been presented with claims or data about politically charged topics, such as the causes or treatment of autism and the harms and benefits of the COVID-19 vaccine. Some people with strong feelings about these topics have a hard time looking at applicable data in a disinterested way. Below is a critical appraisal of a recent article about one of these topics. We have articles coming on Sensible Medicine about similarly charged topics in the coming weeks. I hope you’ll bear with us.

Critical Appraisal

The initial RCT evaluating the COVID was strikingly positive. There was little question about the vaccine’s benefit for unvaccinated and mostly non-immune people. Since the initial trials, we have had to rely on observational studies or trials that used antibody titer endpoints to guide recommendations regarding booster shots. Presently, the FDA is seeking new RCT data for the COVID vaccine in lower-risk people.

The NEJM recently published another observational study of the COVID vaccine. Its novel design adds something to our understanding of the effectiveness of the COVID vaccine in 2025.

Design

The study compared outcomes among veterans who received the COVID vaccine (about 2/3 Moderna mRNA, 1/3 Pfizer mRNA) and influenza vaccines on the same day with those who received the influenza vaccine alone. Enrolment ran from 9/3/24 to 12/31/24. This was kind of a brilliant design as it reduced the risk of healthy vaccinee bias. The researchers compared people who wanted the flu and COVID shot with those who wanted the flu shot but were COVID vaccine hesitant (the researchers excluded patients who received a COVID vaccine, had a laboratory-confirmed positive COVID test, or influenza within 90 days before the date of enrollment). All patients had had at least one COVID shot during winter 2023-2024. Patients in the no-COVID vaccine group were censored if they received the COVID shot at a later date.

The outcomes were the difference in rates of COVID–associated emergency department visits, hospitalizations, and deaths at 6 months – more on this below.

In the end, 164,132 people who received the COVID vaccine were compared with 131,839 who did not. Table I shows that the groups were well matched even before weighting. The mean age of the patients was about 72, 92% were men, 70% were white, 26% black, and about 18% smoked.

Results

The results are summarized in Table 2:

The important findings are the vaccine effectiveness in column 1, which ranges from 28.3%-64%, depending on the outcome, with higher efficacy for worse outcomes. Effectiveness is 1-RR, so the relative risks ranged from 0.36 to 0.72. From the risk difference in column 4, we can calculate numbers-needed-to-vaccinate (NNV). These come out to 546 for a COVID–associated emergency department visits; 1338 for COVID–associated hospitalization; 4545 for COVID–associated death; and 549 for a composite of these outcomes. Of course, these numbers depend on the incidence of infection — the more circulating disease, the lower the NNVs.

Comment

The good: I think this paper yields the best data with which to counsel patients in the “post-immunity” period. I feel pretty good about using these numbers to counsel my patients.

The bad: This is pretty obvious. The endpoints in this study are inadequate. First, I wanted a report of sick days. At this point, most people are, rightly, thinking about getting the COVID shot to keep them from feeling crummy for a few days rather than to keep them alive. I suspect that sick day data would have made the vaccine look even more effective than it does in this study. Obviously, this data was not available to the researchers in this retrospective design.

Second, I suspect that people who got both vaccines were more likely to seek care. This makes the ER visit data suspect.

Finally, I desperately want numbers for all-cause hospitalizations and all-cause mortality. These numbers are critical for counseling patients. In the absence of “sick-day data,” I am unlikely to aggressively advocate for a vaccine that decreases the number of COVID hospitalizations but not hospitalizations. Maybe even more importantly, we are currently faced with real vaccine skepticism. No data will win over the “antivax crowd”, those not vaccinating their kids against measles. Still, robust data will win over people like those included in this study, people who voluntarily got the flu vaccine but not the COVID vaccine. Data showing that the shot reduced COVID death AND overall death would be powerful. Not even commenting on the overall death is, well, deeply unsatisfying.

Conclusion

This study is more than another observational study comparing poorly matched groups or another trial of antibody response. Still, it falls short of an RCT showing the effectiveness of the COVID vaccine today. In this case, I owe this shortcoming less to the observational design and more to the lack of all-cause hospitalization and mortality data.