Designing Studies for COVID Treatments, 9/18
why I would not just take a doctor's word for it
If a doctor reports success with a COVID treatment, should we weigh that evidence more highly than a clinical trial? This question came up in a previous post. My bias is to answer with a very strong “No.”
Of course, a clinical trial may be poorly designed. But it is likely to be more rigorous than a doctor’s experience.
If you are going to do a rigorous study, you have to decide beforehand which outcome to measure. Otherwise, if you track a bunch of different outcomes, you might report the results only on one type of outcome, such as length of hospitalization, when no other outcomes show significant results. This is known as p-hacking.
In the case of COVID, I would choose mortality as the outcome to measure. This outcome has the advantage that it is pretty clear to judge. The patient either died or did not.
But a disadvantage of using death as the outcome to measure is that not very many people die from COVID. If you apply a treatment to 50 patients and none of them die, that may not prove anything about the treatment.
In fact, the dominant factor in COVID mortality is co-morbidities. If a doctor presents evidence that he gave his preferred treatment to 200 COVID patients, and they all lived, the first question to ask is what were the co-morbidities in his patient population. If only a handful had heart problems or extreme obesity, then he is not offering persuasive evidence.
If I were evaluating a treatment using the randomized control trial method, I would choose a modified random sample. That is, I would start with a pool of COVID patients in which people with co-morbidities are over-represented. Then I would divide this modified sample randomly between treatment and control groups.
Relative to the number of people who contract COVID, death is a rare event. (It adds up to a lot of deaths because so many people get it.) That means you need a very careful study in order to demonstrate the effectiveness of a treatment.
Another case where rare events are hard to study is Sudden Infant Death Syndrome. As you know, one of my pet peeves is that parents are instructed to put infants on their backs to sleep. Parents are told that this will reduce the chances of Sudden Infant Death Syndrome. But SIDS is an extremely rare event, much rarer than death from COVID. To study a SIDS-prevention treatment carefully, you would need an enormous RCT, with thousands and thousands of infants in both the control group and the treatment group. No one has done this, nor will they. As a result, I believe that back-sleeping is a superstition. I should note that Emily Oster, who I respect and who is more on top of the research, disagrees with me.
Between 1946 and 1989 the Common Cold Research Unit (CCRU) in the UK spent nearly 50 years and a lot of money investigating the causes, possible cures and vaccines for the Common Cold and Influenza. It used volunteers whom it infected with various viruses and tried treatments and vaccines. During its time it produced 1006 published papers and in 1965 discovered the Human coronaviruses responsible for 10% of Colds - but it never found a cure/treatment or safe, effective vaccines and gave up and closed down.
Are we to believe that suddenly the current celebrated coronavirus - which operates and produces the same symptoms as the Common Cold, except for in 99.5% of those infected much milder - and is distinguished from some other coronaviruses by being rather better at cell invasion, has a treatment regime of readily and long available drugs? If so isn’t it the case that if they work on this coronavirus they should work on the others, and why not all the other respiratory viruses whose route and method of infection and reproduction by cell invasion is the same?
We seem to be caught in a trap, the trap of faux-novelty, that THIS virus is an organism completely new in the Universe - it’s a virus Jim but not as we know it - so unlike any other, the only virus on Earth, causing the only disease on Earth, upon which ‘novel’ vaccines, ‘novel’ treatments, ‘novel’ mask use, ‘novel’ mass, quasi-isolation of healthy people can work, whereas they have failed or don’t work on everything else.
We could consider perhaps that the supposed treatments don’t work on THIS coronavirus for all the same reasons they don’t work on all the other coronavirus and respiratory viral infections. A proper blinded, comparative study using volunteers - why not? the CCRU did that and this is not a fatal virus, no more so than others in the most susceptible - is the only way to decide whether we have at long last discovered a cure for the Common Cold.
Meanwhile I’ll stick with hot toddy and feeling sorry for myself during the 5 days coming, 5 days going period normal for these viruses and keep the chemicals out of my body.
Even the mRNA vaccine trials with 80k people between them could not show a mortality benefit. The same number of people died in the vaccine arm as the control arm* despite the vaccine preventing thousands of infections. If this can’t show a mortality benefit, the effect of Covid on mortality must be pretty small and there would be almost no way to show a benefit for any treatment with even a large effect size, except with a truly massive RCT.
(It is interesting to ask why we choose to ignore some very large RCTs showing no mortality benefit and instead just assume a benefit anyway (eg the vaccine), and yet why we choose to accept other much smaller RCTs showing no benefit as infallible (eg anything other than the vaccine)...)
*eg for Pfizer see table s4 here: https://www.medrxiv.org/content/medrxiv/early/2021/07/28/2021.07.28.21261159/DC1/embed/media-1.pdf?download=true