How well statistics work: a lesson learned from Pfizer Covid Vaccine

Today I deviate somewhat from the financial markets to bring forth a relevant point that equally translates from the medical field. We look at the reported effectiveness of the Pfizer Covid vaccine. Please be advised that I do not have a personal opinion on the vaccine’s effectiveness; however, I merely report what it has been less obvious when reading the government data.  Here I will take you through the process by which the reported effectiveness is derived and how it might not necessarily be obvious from the initial CDC reports.

 The CDC states that “based on evidence from clinical trials the Pfizer-BioNTech vaccine was 95% effective at preventing laboratory-confirmed COVID-19 illness in people without evidence of previous infection.” How was this 95% effectiveness derived? Here is the short answer: by a mere 170 patients! However, you will have to dig, critically analyze written statements, and cross-reference to other medical materials to clearly see that answer.

The point above from the CDC is linked to its Morbidity and Mortality Weekly Report where it explains in further detail how the 95% effectiveness is derived. Specifically, the relevant section (copy/pasted below in italics and in parenthesis) states the following [with my comments in brackets]:

“The body of evidence for the Pfizer-BioNTech COVID-19 vaccine was primarily informed by one large, randomized, double-blind, placebo-controlled Phase II/III clinical trial that enrolled >43,000 participants (median age = 52 years, range = 16–91 years) (5,6).”

[My Comment: 43K seems like a large number of people tested for a vaccine.  My mind thinks: this number of people from where the 95% effectiveness was derived.]

“Interim findings from this clinical trial, using data from participants with a median of 2 months of follow-up, indicate that the Pfizer-BioNTech COVID-19 vaccine was 95.0% effective (95% confidence interval = 90.3%–97.6%) in preventing symptomatic laboratory-confirmed COVID-19 in persons without evidence of previous SARS-CoV-2 infection.”

[My Comment: Here is the first catch: how many “symptomatic laboratory-confirmed COVID-19 in persons without evidence of previous SARS-CoV-2 infections” are we talking about? If they clearly call out this group, it must mean that the 43K sample noted previously included people who were infected or had been infected, or were not able to be determined. Nowhere in the CDC article are we told the number of “symptomatic laboratory-confirmed COVID-19 in persons without evidence of previous SARS-CoV-2 infections”]

“Consistent high efficacy (≥92%) was observed across age, sex, race, and ethnicity categories and among persons with underlying medical conditions.”

[My Comment: Presumably, this is representative of “symptomatic laboratory-confirmed COVID-19 in persons without evidence of previous SARS-CoV-2 infections”.]

“Efficacy was similarly high in a secondary analysis including participants both with or without evidence of previous SARS-CoV-2 infection.”  

[My Comment: Here is another clue leading us to conclude that the 43K sample included a mixed bag of people who were exposed to COVID. But, again, nowhere in the CDC article are told of this breakdown.]

With that background and question at hand, namely, how many “symptomatic laboratory-confirmed COVID-19 in persons without evidence of previous SARS-CoV-2 infections” are there, we went to The New England Journal of Medicine where they clearly state that “[t]here were 8 cases of Covid-19 with onset at least 7 days after the second dose among participants assigned to receive BNT162b2 and 162 cases among those assigned to placebo; BNT162b2 was 95% effective in preventing Covid-19 (95% credible interval, 90.3 to 97.6).”

When you divide the 162 who did not get COVID by 170, and then multiply by 100, you then get the 95% vaccine effectiveness that is being widely reported. It is not, as you might be led to conclude, based on the 43K total being reported in the CDC article.

Lesson Learned: Always check the data and the critically think through categorical statements being made.