Magic Cutoffs

A famous podcaster I'm not quite brave enough to name has a habit that drives me crazy. He'll say things like:

"According to this study, eating six goji berries will improve your sleep. So, people who want to sleep well, try eating goji berries, but in order to get this impact remember you have to eat six or more."

Put aside the now-infamous issues with many such studies: they may have been done on 15 undergrads, they may have been p-hacked, the data may have been literally made up.

But even if the study itself was gold-standard excellent, you'd need to have very specific conditions to think that six berries specifically is a magical cutoff, and there's no effect below that number, and a full effect above it.

As a first pass, it might be better to assume – if you have no other knowledge of a given process – that effects are a linear output of the input. That is: if the study truly showed that eating six berries made people sleep for one additional hour, then eating three berries would make them sleep an additional half-hour, and one berry would make them sleep an 10 additional minutes.

But actually, for many processes, it makes more sense to assume diminishing returns: maybe the first berry accounts for 50% of the impact, the second for another 25%, and the sixth berry is contributing almost-nothing.

In fact, the researchers might have used six berries in the study because it was such a high number that they felt confident that it would cause the desired result, while knowing that a much smaller intervention would also most-likely do the trick.

Most studies are not checking the impact of 0, 1, 2 ... 5, and 6 berries, and then telling you 6 is where the impact is; rather, they're checking an intervention against no intervention, and so their incentive is to make the intervention big enough that they're confident they'll get a result.

Goji berries are both small and delicious, so there's not really an issue with eating 6 of them when you only needed 3. But many of the interventions encouraged in this fashion are accurately quite costly. E.g. doing two hours of heavy weightlifting is a lot harder than doing one hour of lighter weightlifting, so if the latter gets you most of the impact then for many people it's a better choice. Or, fasting for 18 hours a day is much harder than fasting for 12 hours per day, so it's good to know if the easier intervention gets you most or all of the benefits.

The world is full of weird wonders, and there are certainly cases where a given process creates no impact with n-1 inputs, and suddenly has a huge impact once you hit n. But in most cases, there's not a magic cutoff at the level of intervention a researcher chose to study, and (in fact, by contrast) you can often get the vast majority of the impact with far less of the input.

Here's the thing: the belief in magic cutoffs can actually have vast societal implications. The most recent, most serious example was probably the implementation of the Covid vaccine.

Here's how I understand it: when the vaccine companies were running trials on their new vaccines, they were (understandably) under immense pressure to make sure the vaccines passed all trials on their first try. As a result, the size of the doses was relatively high: like the imaginary researchers I described above, their aim was to choose a (safe) dose size that would certainly create a required effect, rather than finding the smallest dose size that would get even 99% of the same benefits.

Once the vaccine came out, various researchers (led by economist Alex Tabarrok) found that a "half" dose of the Moderna vaccine conferred almost as much protection as a full dose. Given that the world was critically short on vaccine supply at the time, there was a very strong argument that we'd be better off vaccinating two people with 50µg each than one person with 100µg.

I can't speak to what determined the public policy response, or what other factors the medical authorities had to consider. But just in terms of public reaction, I sense there were a lot of people who treated the "full" dose as a Magic Cutoff, and were distrustful of the very idea of smaller doses, regardless of evidence about their impact.

And this truly seemed to have an element of magical thinking. The different pharmaceutical companies made highly contingent choices about dose size and schedule, but once the trials were run those contingent dose sizes were considered as Magic Thresholds. That's why many people considered someone who'd had one shot of J&J "fully vaccinated" while considering someone with a single Moderna dose "not fully vaccinated", even though just the first Moderna seems to have conferred more protection than the first (and only) J&J dose.

All of this seems rooted in Magic Cutoff thinking. You might not be able to affect national vaccine policy, but at the very least you can control your exercise patterns and goji berry consumption.