I was intrigued by this 10% claim, since I’m an admitted gene science skeptic and the authors were suggesting that autism gene science is making great progress: 10% down, 90% to go. So I started a search for the Original Source Of The Estimated Number cited by Weiss et al (the OSOTEN for short), thinking that such a strong statement of progress on the part of genetic science ought to be backed up by some pretty convincing evidence. As readers of that essay will remember, I followed the direct citation trail provided by Weiss et al and came up with the remarkably little evidence of any analytical foundation whatsoever for the OSOTEN. Instead, all I found was evidence of a completely unsupported “chain of collective reasoning”, one in which a 2001 assertion by Susan Folstein and Beth Rosen-Sheidley (who said that it was “generally agreed” that 10-15% of autism cases were attributable to “an identifiable Mendelian condition or genetic syndrome”) got picked up eagerly and passed along uncritically by scientists, taking on a kind of social momentum as it passed from paper to paper. Because the citing scientists were so eager to project a firm foundation of support for genetic causation, they seemed not to take special care to examine the foundations of one of their key arguments.

Reflecting on the remarkable longevity of this unsupported claim, I pointed to a fascinating study by Andrey Rzhestky showing how such faulty “chains of collective reasoning” might be pervasive in science. In a fascinating study of the way in which statements about molecular interactions can gain this kind of social momentum among scientists, Rzhetsky and his colleagues reported a surprising result.

We…found that previously published statements, regardless of whether they are subsequently shown to be true or false, can have a profound effect on interpretations of further experiments and the probability that a scientific community would converge to a correct position.

Recognizing that false but influential citation trails often turn up in multiple places and not simply in the place one first observes the terminal point of a single trail, I’ve remained on alert for other possible “chains of collective reasoning” regarding the 10% number. Since it’s entirely possible, even likely, that the first citation trail one follows in chasing down a scientific fact may lead in the wrong direction, I’ve been looking for other trail markers that might lead to a more successful path to the OSOTEN, markers that might (or might not) back up the 10% claim.

Earlier this month, I found one. In a January review paper entitled “Genetics evaluation for the etiologic diagnosis of autism spectrum disorders”, two autism geneticists from Omaha, Nebraska named Bradley Schaefer and Nancy Mendelsohn made the following statement.

The generally reported range for identifying a diagnosis [in a comprehensive genetics evaluation] in persons with autism is 6–15%. Recent reports of the yield of a genetics diagnostic evaluation for autism range from 10% to 40%.

Wow, 10-40%?! Could this be the clue I needed to find the elusive OSOTEN? Perhaps Weiss et al had simply chosen the wrong branch on the OSOTEN citation tree and this new article by Schaefer and Mendelsohn might launch a more fruitful search. Perhaps now I could find and critically assess the evidence base for this “generally accepted” estimate that 10% of autism cases (indeed maybe more!) are explained by genes alone. Armed with the reference list from this new article, I went citation hunting again.

The first stop was quick. In Bradley and Mendelson’s passage quoted above, there were two studies cited, another one from Nebraska written by the same Dr. Schaefer and another colleague, and a second one by a genetics group from Columbus Ohio. I downloaded each of these papers and read them. As any citation hunter knows, the best place to look for clusters of evidence is in the introductions and discussion sections of related scientific papers and I was not let down as I scanned these two. After a lengthy and fruitless search for the OSOTEN, I found myself confronted with a new abundance of evidence: ten citations, all with what looked like some real analysis on something called “diagnostic yield.”

I realized I had hit the mother load.

As it turns out, these ten studies comprised a hefty and recent body of literature on autism by clinical geneticists, most of them published in just the last three years. Many of the study team members were geneticists by trade, but not the theoretical types that publish genome scans in The New England Journal of Medicine. Instead, this body of work comes from autism doctors with a clinical bent, many of whom conduct genetic testing as part of their daily work with populations of children. Not surprisingly, these doctors tend to use more practical language. In all of the studies, the authors used a clinically-oriented term for the search for autism genes in real people, a search where their goal is to maximize their yield of genetic diagnoses in their autistic patient populations: hence the term “diagnostic yield.” 

Following this new citation trail was quite an educational experience, as much for the attitude of the researchers involved as for their results. The attitude of these clinicians was exemplified by the January 2008 paper by Schaefer and Mendelsohn, a review paper in which these two Omaha geneticists were clearly looking to encourage referrals of autism cases to genetics clinics everywhere.

While the logic for an evaluation should seem intuitive to a clinical geneticist, the rationale for proceeding with diagnostic testing may need to be reviewed with the family and with referral sources. It is our contention that all patients with autism be offered a thorough diagnostic evaluation.

In their way, they were simply offering up a bit of professional advertising for themselves and their colleagues. This wasn’t particularly surprising, of course, but it wasn’t a concern of any consequence either. The real question was, what did their data show? In reading through the ten studies, I did find traces of the same confident tone, although several research teams wrote with a more measured voice than the Omaha duo. But I was also not surprised to find far less support for the 10% number than the “chain of collective reasoning” would have led a less skeptical observer to expect. And as I read more carefully, I found that the confident foundation of the 10% genetics claim crumbled under close inspection. The analysis was a bit technical, so the crumbling is best understood by giving a summary overview of the findings of the ten studies, which really fell into three distinct groups.

The first group, the one that included the two studies cited in Schaefer & Mendelsohn’s review, was by far the most optimistic. Four different genetics clinics (one each from Omaha, Columbus Ohio, Stanford University and the University of Manitoba) had gone back into their patient records to report on their genetic analyses of autism cases. With study samples ranging from 32 to 91 patients, these authors reported an overall success rate—what they all called a “diagnostic yield”—that ranged from 8 to 41%. Summing the results across all four studies, the overall sample of 278 patients showed a reported yield of nearly 15%. If this were indeed a sustainable yield, perhaps the 10% claim was even conservative when held up against the evidence in these four OSOTENs.

The problem was, the claims were not conservative at all, because both the study samples and the success rates were misleading. First, all four of these studies were conducted by genetics clinics working with referrals, not a random sample of the autism population. As one of them confessed, “there may have been a bias in referral of selected families to the genetics clinic, which may have raised the proportion of various associated disorders.” Second, since all of the studies were retrospective, all were subject to both selection bias and publication bias: selection bias, because the authors were motivated to design studies that selected patients who would boost their yield; publication bias, because clinics with more negative results would have less incentive to write up their findings.

But more importantly, three out of the four studies, when reporting their diagnostic yield significantly overstated their genetics yield. Out of the 41 cases that the authors claim to have “explained”, only 31 of them had measurable genetic markers. Recalculating the diagnostic yield based on 31 genetics diagnoses reduced the yield to 11%, far lower than the 15% reports and nowhere near the most optimistic 40% claim.

But still, even an 11% finding would be a valid OSOTEN, if it were credible. And here’s where the studies revealed, quite literally how fragile the foundation of the 10% claim was. Of the 31 cases that made up the successful genetics yield of the clinics’ autism sample, over half of them carried just two diagnoses: Rett syndrome and Fragile X syndrome. But when you place all the scientific hubris to the side and consider this simple finding, there’s just no way that these two diagnoses could explain anything close to 10% of autism cases. Since by definition, Rett syndrome occurs only in girls, and most studies place the frequency of Rett cases among girls at about 1 in 20,000, there’s no possible way that the high rate of Rett syndrome (nearly 5%!) in these samples was representative of the larger autism population. In any reliable sample, the Rett rate would be vanishingly small.

By contrast, the Fragile X findings in this group, about 1 ½% of cases, were more in line with at least some recent estimates. The estimates of how frequently the Fragile X genetic mutation is present in autism (which is actually an autism susceptibility gene, since many Fragile X cases don’t have autism) vary all over the map, but the more reliable published estimates generally fall in the 1-2% range. But because of the pervasive issue of referral bias, it’s likely that even these estimates may be an overly optimistic. An experienced autism clinician recently made the following comment on a private list.

In almost every audience in which I speak I ask how many people have done a Fragile-X test. About half the audience raises their hands. When I ask how many were positive, the most common number was zero. About 10% of the time, one person is positive. I have never seen two people raise their hand. [Geneticists estimate] that 5% would be positive for fragile-X if more screening were done. I estimate that the incidence is 0.3%.

Whether the rate is 0.3% or 1.5% is hardly material. The point is simple, if the best evidence for genetic yield that the most optimistic authors can find is Fragile X syndrome, then entire basis for the evidence of diagnostic yield is fragile indeed.

So much for the first group. In the second group of studies, three research groups (one from Missouri, one from Israel and one from the Mayo Clinic) didn’t report being part of a genetics clinic but had a similar retrospective design, with all of the potential selection and publication biases that kind of design would entail. These three studies had larger samples, ranging from 94 to 182 cases, and they reported lower genetics yields, from 3.3% to 6.5% (the lowest yield came from the Mayo Clinic group). Overall, their average yield was 5%, with well over a third of these coming from Rett and Fragile X syndrome diagnoses.

Clearly, the foundation for the 10% number was looking more fragile all the time.

Finally, in the third group, there were two papers that had the kind of design you would expect from an unbiased study. They were prospective, in that they selected their sample from a general population before they did their genetic analysis, and they weren’t working with a clinical genetics patient load. In two relatively small samples, one group had a 5.9% genetics yield in an Italian patient base, the other from Ontario had a 0% yield.

So at the end of my latest citation hunt, what lessons did my search for the OSOTEN teach me this time? When I sum up the statistical findings, the numbers themselves were pretty clear.

1. There is no valid foundation for the claim that 10% of autism cases can be explained by genetic causes. The claim appears to be just another case of a faulty “chain of collective reasoning” and, despite being “generally accepted”, it is specifically wrong.

2. Even when there was documented evidence of a positive diagnostic yield, the evidence was overstated. The overall genetics yield in all of these studies came to less than 7%, and well under 6% if you exclude the single study that claimed to have explained 30% of cases.

3. When you take out known rare forms of autism like Rett and Fragile X syndromes, the reported yields collapsed even further. Nearly half of the foundation of success in generating a diagnostic yield rested on these two exceedingly rare diagnoses.

But the most important lesson came less from the data than from the tone of the researchers. When you read the papers written by member of the autism genetic science community, one thing is blazingly obvious. These people are selling. And these studies are basically just pitches for the money that they hope our children can bring them.

They’re pitching their clinics to their referral sources; they’re pitching their research ideas to the grant-makers; and they’re pitching their latest diagnostic protocol to their colleagues. Most of all, they’re pitching the grand experiment of explaining human diseases with genes. And because it’s a pitch that benefits major scientific constituencies, most scientists simply allow the pitch to pass by without challenge. After all, everyone benefits when they make another sale.

Everyone, of course, except our kids.

Which, of course, is why we all can’t be satisfied with the social momentum of science and the fragile evidence base on which critical intellectual foundations rest. We all need to learn how to be critical consumers of faulty chains of reasoning. And to stop them cold in their tracks.

Mark Blaxill is Editor at Large of Age of Autism He also gratefully acknowledges the original source for the idea behind this essay, a wonderful book by Robert Merton titled On the Shoulders of Giants (affectionately known to its admiring readers as OTSOG).

References (for the scientifically minded):

Review paper
Schaefer GB, Mendelsohn NJ. Genetics evaluation for the etiologic diagnosis of autism spectrum disorders. Genet Med. 2008;10(1):4-12.

Group 1
1. Schaefer GB, Lutz RE. Diagnostic yield in the clinical genetic evaluation of autism spectrum disorders. Genet Med. 2006;8(9):549-56.

2. Abdul-Rahman OA, Hudgins L. The diagnostic utility of a genetics evaluation in children with pervasive developmental disorders. Genet Med. 2006 Jan;8(1):50-4.

3: Herman GE, Henninger N, Ratliff-Schaub K, Pastore M, Fitzgerald S, McBride KL. Genetic testing in autism: how much is enough? Genet Med. 2007;9(5):268-74.

4. Chudley AE, Gutierrez E, Jocelyn LJ, Chodirker BN. Outcomes of genetic evaluation in children with pervasive developmental disorder. J Dev Behav Pediatr. 1998;19(5):321-5.

Group 2
1. Battaglia A, Carey JC.  Etiologic yield of autistic spectrum disorders: a prospective study. Am J Med Genet C Semin Med Genet. 2006;142(1):3-7.

2. Challman TD, Barbaresi WJ, Katusic SK, Weaver A. The yield of the medical evaluation of children with pervasive developmental disorders. J Autism Dev Disord. 2003;33(2):187-92.

3. Kosinovsky B, Hermon S, Yoran-Hegesh R, Golomb A, Senecky Y, Goez H, Kramer U. The yield of laboratory investigations in children with infantile autism. J Neural Transm. 2005;112(4):587-96.

Group 3
1. Shevell MI, Majnemer A, Rosenbaum P, Abrahamowicz M. Etiologic yield of autistic spectrum disorders: a prospective study. J Child Neurol. 2001;16(7):509-12.

2. Miles JH, Hillman RE. Value of a clinical morphology examination in autism. Am J Med Genet. 2000;91(4):245-53.