Better in their words…

A new paper is out in JAMA Network Open and is making the rounds. It makes the claim that as this ratio increases:

….mortality among black people falls. The press and many commenters uncritically accept the conclusion. Headlines read:

• But is the paper accurate?
• Does it show a causal link?
• If a higher fraction of PCPs are black doctors would black life expectancy rise?

Here, I will critically evaluate the paper, and raise a central concern: In a well intentioned effort to promote social causes, if we lower our standards of evidence, we are going to shoot ourselves in the foot: undermining those causes, giving fodder to opponents, and sabotaging critical thinking more broadly.

I think that we must hold all scientific claims— whether it is about ivermectin, masking or whether more doctors of any demographic improve patient outcomes— to the same standard: aka, is this work good. Before I turn to the paper and probe that question, I want to briefly comment on the policy goal.

The policy goal

This paper is in service of a broader policy agenda item. Should the United States do a better job in training and retaining Black doctors born in the United States? This is an important policy question, and my verdict is yes. The strongest argument in favor of reforming the system to achieve this is: if public education were equal for all kids born in the US— fair across districts and locations— then there is no reason to think that the fraction of black doctors would not be similar to the fraction of black people in America. In other words, disparities in the number of black doctors likely reflects inequity in the entire educational pipeline and this is worth addressing in and of itself. America is about equality of opportunity. I believe this is true, and politicians should act. I would place the bulk of those efforts on innovated early life programs/ k-12 education, and I would run RCTs.

Do some doctors save more lives?

The paper and commenters make the argument that if black doctors save more lives then that is an additional reason to pursue this goal. I think this line of reasoning is misguided and dangerous. What if white doctors save more lives than Indian doctors? What if Korean Americans save more than Japanese Americans? One can imagine many unpalatable findings from a churning body of low quality observational research with countless analytic plans.

Second, this is all based on group characteristics, but surely there will be more variation within groups than between groups. What if Tom and Aspen save more lives than Suzy and Jasper? What if Jasper is in the bottom 10% repeatedly? (In retrospective observational data). What if were review Jasper’s charts and cannot find even a single decision we disagree with, but consistently see poorer outcomes? Is it time for Jasper to move to Pharma? Maybe Ashish Jha can make an introduction.

The key lesson here is low quality data (as I will show this is) can find things we like, but also things we don’t like, and I don’t think we have thought through the implications

The null hypothesis in health care is any one doctor doesn’t matter that much

A higher black representation ratio saves a lot of lives! An implausibly large effect. For every 10% increase in the representation ratio, life expectancy goes up for Blacks by 30 days— even if the black doctors are not caring for the black patients— just because they exist in the county! Is this plausible?

The average gain from taking statins in pooled analysis is on the order of days (3-7), and the effect seen here is comparable to 27 days seen in the 4 S trial from taking 5.8 years of statin after unstable angina or MI. This is among sick people who are at super high risk and when they personally take a pill for years in a trial.

Now you are saying a representation ratio going from 0.6 to 0.66 has the same effect even if there is no evidence that this results in more black doctor- black patient pairings? The effect size is ludicrous. It cannot possibly be true.

The truth is that in the entire literature that some doctors are better than others is threadbare, and the null hypothesis should be— since so many doctors touch a patient, since so much of health the downstream product of lifetime of socioeconomics, genetics, and environment, since most of medicine has modest effect sizes, since compliance is always less than perfect, since similarities in practice outweigh differences— it probably doesn’t matter. If you find a difference by gender, race, IMG status, or whether it was Tom or Jasper— you are probably finding noise or confounding, rather than a real signal. And yet, if the paper reaches the preferred conclusion— this common sense is ignored— and people tweet breathlessly.

The paper in question

Now let us turn to the paper in question, we have already highlighted 2 problems:

1. Effect size is preposterous
2. No proof that these black doctors are caring for black patients

But it doesn’t stop there.

3. Race is defined from self reports. But not all racial groups self report equally, and how do I know it doesn’t vary by geography/ region/ county? If blacks always self report more or less than whites and that is constant across regions and times, it would be ok, but how do we know that. That might not be true.
4. Why is the exposure: this convoluted ratio. Why not just # of black doctors per capita, or # of black doctors per capita black people? Did the authors preregister this ratio? Did they try other things before settling on this? How many other groups and how many other analytic plans exist? According to this ratio, if I fired 10% of the white doctors, I would improve life expectancy for blacks. Bizarre.
5. The model is complicated, but my reading (tell me if I am wrong) is that the mortality changes seen here arose between 0 and 10 years. Given the data is coded in only 3 times (and each 5 years is time 0,1,2,) we expect the ratio in 2019 to reflect the immediate impact on life expectancy, and the ratio in the 2 prior times to work over 5 and 10 years max. How do PCPs impact survival so dramatically and so quickly? And doesn’t the inclusion of 2019 ratio and outcomes imply we are also looking for immediate effects.
6. Because of the choice of the ratio, the authors have to discard 50% of data b/c they could not find even 1 self reported black PCP in a county. How would the data look, if these points were included and the exposure was # black PCPs/ per capita or per capita black people? Also this to me is the biggest finding in and of itself! No need to gild the lily and offer a dubious association.
7. Adjusting for poverty rate is not the same as adjusting for SES.
8. Analytic flexibility is astronomical , “after testing several models for the level 1 residuals (eg, homoscedastic, autoregressive error structure, etc), mixed-effects growth models with an unstructured residual covariance matrix were used (1) to regress life-expectancy, age-adjusted all-cause mortality rates, and a log-transformed measure of mortality rate disparity between Black and White individuals on the log-transformed representativeness ratio within each county”. Analytic plans for things like this should be pre-registered.
9. Not a criticism, but honest q: Can someone explain this figure to me. If the independent variable is .69, .77, and .85, and you log transform, how do you get this scale from 0 to 10? (Also some raw values must be over 1)

10. There is no data supplement. Wtf.
11. The way PCP is coded— isn’t it likely some subspecialists are miscoded as PCPs? How do I know this doesn’t vary by geography, location and race?
12. What about counties with more vs. less cross county health care receipt? Counties adjacent to large cities, etc? Does this not vary by geography and time?
13. How we do we know direction? Do more black physicians as fraction of all physicians standardized for blacks in the population improve black life expectancy? Or people (richer/ healthier) retire and move to regions that culturally suit them, and have doctors they prefer?

There are a million other points I could make, but it would get so technical, the reader will want to bludgeon me. Let me say, I have zero idea from reading this paper if this association is true or causal. My intuition tells me that black representation is just a marker of more desirable, socially diverse, thriving, growing places to live, and little more. I have zero confidence that the PCPs here are improving these outcomes through their medical practice.

Uncritical tweets and coverage

It is sad that we are descending into a world, where “purity” is prized over truth. As I said at the start, I share the policy goal of the promoters of this paper, but I think it is a terrible, awful idea to use low quality work like this to make ridiculous claims. First, those who disagree with the policy goal— at least the few smart ones— can carve this paper to bits. Second, it just further encourages an attitude in academics where doctors get dumber. Why did we mask 2 year olds, why do people say equipoise doesn’t exist for mask trials, and why didn’t we run RCTs of boosters— b/c many of us are not smart! Encouraging blind acceptance of low quality analyses— as this paper is— because it aligns with our political side further fuels our stupidity. It might be the root of all evil.

I am aware of one small, single region randomized trial showing black vs. asian/white docs improve adherence to 6 measures, and I will return someday to critically evaluate that, but no one has ever shown that race concordance can improve hard outcomes to this magnitude in prospective fashion, and remember this study does not even ensure race concordance, i.e. we have no idea if the extra black PCP ratio is seeing black patients. Likely it is just a marker as I describe.

Of course, this is a paper on a hot topic, and simple minded people will try to turn my essay in a binary comment, but I am a real scientific progressive. That means while I share your goal— I think p hacking a dataset to yield a clearly bullshit conclusion doesn’t help anyone. It may lead to a some uncritical headlines from reporters who forgot what journalism means, and you will preach to the choir, but you are not advancing the cause. PS: the stat article doesn’t even summarize the exposure correctly.

The path forward is always and only pristine science. That should be something we all agree on.

Here is the paper: