The Lightning Test That Rewrote What Counts as Long COVID

Hamed Nilforoshan went looking for an outcome long COVID could not possibly cause. He had spent months reproducing the results of high-profile studies that tied the virus to an unusually broad list of lasting symptoms, and when he used their approach he matched their findings exactly. Then he asked the same approach a question with only one honest answer, and it failed. Lightning strikes, dog bites and gunshot wounds all showed up as effects of long COVID.

Tested people and untested people were never alike to begin with. Earlier studies had set people who tested positive for COVID against people who were never diagnosed, a group that included people who were never tested at all. People who seek a test may already carry other conditions, or be ill with something else. They may also see doctors more often and collect more entries in their medical records. Every one of those differences lands in the comparison as if the virus caused it. “Even with 244 million patients, you can get highly precise answers to the wrong question,” Jure Leskovec, a co-senior author of the new study and a professor of computer science at Stanford, said in the university’s release. The release says the study confirmed that tested people differ systematically from those who do not.

The Stanford team rebuilt the comparison from the test itself. Only patients at their first PCR test for COVID entered the study. Those who tested positive were set against those who tested negative, so both sides had walked into a clinic for the same reason. Their past claims histories were aligned with a synthetic-control design borrowed from economics, a method that builds each comparison from the recorded histories of similar patients before looking at what happened next. The virus stayed the same. Stefan Wager, a co-senior author and an associate professor at Stanford Graduate School of Business, said it was exciting to see the design’s promise applied to a large-scale medical study, and that he hoped it would be used more widely on health data.

With the old comparison restored for a moment, the team reran the impossible-outcome check. This time the impossible outcomes stayed quiet. Firearm injuries, which the unrestricted design had attributed to COVID at a high rate, were not flagged. Adult cerebral palsy, used as another negative control, showed how much positive confounding the old design had carried. The same bias feeds the plausible-looking list and the impossible one, so a method that fails the check cannot separate the two. Leskovec stated the test in the release: “A reliable method should not identify those outcomes as the effect of long COVID.”

The comparison ran across 614 clinical outcomes. The team followed them for two years in 14.4 billion insurance claims from 244.7 million patients in the United States. Between 30 and 119 days after infection, the test-based design attributed 43 outcomes to COVID against 262 under the unrestricted design, 83.5 percent fewer. Between 120 and 359 days, the count fell from 286 to five, 98.2 percent fewer. Between 360 and 719 days, it fell from 369 to zero. The unrestricted design produced the larger count in each window.

Myopathy, muscle disease, carried a relative risk of 3.1 in the early post-acute phase. The figure sounds large until it is set against how uncommon the condition is at baseline. In monthly claims, the absolute excess was modest. Non-TB pneumonia, respiratory problems and a collapsed lung, pneumothorax, were among the other effects the team verified. Elevated risk for myopathy persisted into the 120-to-359-day window alongside systemic problems, but the absolute differences had fallen close to baseline by then. Post-infectious fatigue stayed statistically detectable in that window largely because fatigue is so common in the general population that even a small excess shows up in data this large.

By the third window, every evaluated condition’s risk ratio had dropped below 1.1. No outcome met the paper’s significance threshold after Bonferroni correction. Bonferroni correction tightens the bar when hundreds of outcomes are tested at once. In insurance claims, with this comparison, nothing could be attributed to COVID one to two years after infection. It does not say that nobody still feels ill. The authors write that symptoms beyond one year are not shown to be non-genuine, only that their attributability to COVID is uncertain. The paper’s February preprint put the opposite risk, that the stricter design modestly underestimates some real effects through the same selective testing, at 10 to 20 percent.

Insurance records hold diagnoses and billed care. The authors caution that they cannot capture every patient experience, particularly fatigue and cognitive difficulties that may go undiagnosed or unrecorded. A symptom that never reaches a claim never reaches the count, however real it is to the person living with it. Leskovec said the results should complement physicians’ clinical assessments rather than replace them, and that earlier studies might need revisiting with better controls. The study’s windows also cover both working definitions of the condition. The World Health Organization puts post-COVID-19 condition at usually three months from onset, with symptoms lasting at least two months. The Centers for Disease Control and Prevention puts post-COVID conditions at four weeks or more.

Steven Goodman, a professor of epidemiology and population health at Stanford, read the narrowed picture differently. Goodman said the findings indicate long COVID “may be more similar to other extended post-viral syndromes than many have thought.” Most earlier long-COVID studies used the unrestricted comparison, comparing test-positive patients with controls without a known infection. Nilforoshan’s team reproduced those studies’ findings with the old approach before the impossibility check broke them. Researchers from the University of California, Berkeley, and ETH Zurich also contributed to the work.

The paper appeared in the Proceedings of the National Academy of Sciences, volume 123, article e2604414123. Goodman is also a member of the Stanford Causal Science Center and directs the university’s Program on Research Rigor and Reproducibility.