YOU’RE INThe studies that never reached the shelf
Say helloAdmin Login
2 November 2026 / Aadvik Agastya

The studies that never reached the shelf

In 1979, the psychologist Robert Rosenthal gave a problem in research an unexpectedly ordinary setting: a file drawer. The published literature might contain studies reporting striking effects, he argued, while studies that found no statistically significant effect remained out of sight. A reader could examine every article on the shelf and still misunderstand the evidence, because the shelf was not the whole collection.

The image has stayed with me. A drawer is not a dramatic place. It is where papers go when there is no reason to display them. Yet Rosenthal’s 1979 paper on the “file drawer problem” raised a question larger than the furniture: what happens to our confidence when the results we can inspect have been selected partly because they are interesting?

I find this more unsettling than an outright falsehood. A false claim can, at least in principle, be confronted with a contradictory fact. Here the visible claims may be honestly reported. The distortion lies in the arrangement of what becomes visible. A collection of true statements can leave a false impression if the rules for entering the collection favour one kind of outcome.

Consider a deliberately imaginary example. Ten careful studies ask whether a particular intervention helps with a particular problem. One produces a persuasive estimate of benefit; the other nine produce uncertain or unpromising results. If only the first is written up for a broad audience, its finding has not changed. What has changed is the reader’s sense of how often the effect appears and how much confidence it deserves. The missing studies are part of the question even though they are absent from the answer.

There is an important caution in that example. “No statistically significant effect” does not mean “proof that nothing happened.” A small study may be unable to distinguish a modest benefit from no benefit at all. A badly chosen measure may miss the change researchers hoped to detect. An inconclusive result can be genuinely inconclusive. The problem is not that every uncelebrated study secretly contains the truth. It is that we cannot judge the visible successes properly when we do not know what else was tried.

This changes how I think about the word evidence. We often imagine evidence as an object: a paper, a number, a quotation, a graph. But evidence is also a history of decisions. Which questions were considered worth asking? Which outcomes were defined as important before the work began? Which measurements were reported? Which attempts were submitted, accepted, cited and eventually repeated in books? By the time a claim reaches a reader, it has travelled through many doors.

Medicine makes the stakes unusually clear. A person deciding whether to take a treatment is not asking whether someone once observed an encouraging result. They are asking what the best available evidence suggests about benefits, harms and uncertainty. If disappointing trials disappear, the apparent balance can tilt even when each surviving paper is technically accurate. That is not merely a problem for researchers’ reputations. It can alter decisions made by people who never see a journal.

In September 2004, the International Committee of Medical Journal Editors issued a statement requiring prospective registration for trials seeking consideration at its member journals, with the policy applying to trials beginning enrolment after 1 July 2005. The underlying idea was disarmingly simple: record the existence and intended questions of a trial before its results are known. A study should not become visible only after someone discovers whether its outcome makes a good story.

Registration does not turn a study into a good study. Nor does it, by itself, make its findings public. The committee’s own guidance distinguishes the obligation to register from the separate issue of reporting results. But a dated record can make a later comparison possible. We can ask not only what investigators say they found, but what they said they were going to look for.

That difference matters because research can contain many possible outcomes. Imagine a trial designed primarily to measure whether a treatment reduces a defined symptom after six months. It may also collect information about sleep, appetite, activity and other changes. Those additional observations may be valuable. But if the main result disappoints and a secondary result is later presented as though it had always been the central question, the reader receives a cleaner story than the research actually followed.

This is not an accusation against every change of plan. A study can reveal an unexpected pattern worth pursuing. Science needs the freedom to notice what it did not predict. The obligation is to tell the reader which questions were specified in advance and which arose after looking at the data. Discovery and confirmation are both valuable; they become misleading when one is dressed as the other.

The concern is not hypothetical in the wider research record. A 2015 systematic review in BMC Medicine examined studies comparing registered trial outcomes with published reports. Across the studies it reviewed, the median proportion of trials with a discrepancy in the primary outcome was 31 per cent. The reviewers also stressed considerable variation and moderate-to-high risks of bias in the underlying studies. That figure is not a universal rate for all clinical trials. It is a reason to take the comparison between plans and publications seriously.

The response has developed beyond registration. In 2016, United States health authorities issued a final rule on registration and results submission for certain clinical trials; related National Institutes of Health policy took effect in 2017 for specified NIH-funded work. These requirements have scopes, exceptions and implementation details. They should not be mistaken for a guarantee that every study everywhere is reported. Their significance is that a result can be made publicly available without waiting for a journal to decide that it is exciting.

There is a useful distinction here between making information available and making it trustworthy. ClinicalTrials.gov explains that submitted results pass through quality-control processes, but the existence of a posted record is not equivalent to an independent journal review or a guarantee that every underlying measurement is correct. Transparency gives scrutiny something to work on; it does not do the scrutinising for us.

A different response changes the order in which publication decisions are made. In the Registered Reports format described by Cambridge University Press, a research question and proposed method can be reviewed before the data are collected. An in-principle acceptance commits a journal to consider the completed work on the quality of its method and adherence to the agreed plan, rather than on whether the result turns out to be dramatic. The outcome is no longer supposed to be the audition.

There is a counterargument worth taking seriously. Publishing every failed attempt in full would produce its own kind of noise. A poorly designed study cannot be rescued by being negative. A result that tells us little may not deserve a long article, and repositories require money, maintenance, responsible handling of personal data and clear descriptions if anyone is to use them. An archive that no one can interpret is not automatically a service to knowledge.

The answer, then, is not to treat the file drawer as a warehouse that must be emptied without judgment. It is to make the record of inquiry proportionate and intelligible: register consequential questions, report outcomes that bear on them, explain departures from the plan, preserve appropriate data when it is ethical to do so, and say plainly when an estimate is too uncertain to settle the matter. We can insist on completeness without pretending that all pieces of information have equal weight.

I am also struck by the ethical dimension of this problem. Clinical trials ask people to contribute time, accept inconvenience and sometimes face risk so that something may be learned. The World Health Organization’s 2015 statement on public disclosure treats the selective disappearance of results as a serious concern. Participants cannot reasonably be promised a favourable outcome. They can reasonably expect that an unfavourable or inconclusive one will not be rendered worthless simply because it is difficult to publish.

This is where the subject moves beyond laboratories and enters the life of a reader. A book about an important discovery usually begins near the moment when an idea finally worked. The pages before that moment may contain years of abandoned approaches, measurements that contradicted a favourite explanation and questions that were asked in the wrong form. A lucid account has to select. No writer can reproduce an entire research history. But selection becomes distortion when the finished story makes success look inevitable.

I do not think every popular book needs a catalogue of dead ends. Such a book might be unreadable. What it owes us instead is a sense of the resistance that an idea met: what would have counted against it, what alternatives were seriously considered, and how much of the conclusion survives when the most convenient examples are removed. An explanation becomes more interesting, not less, when we can see the work it had to withstand.

Something similar can happen when we look backward at culture and history, though the comparison has limits. Surviving documents tell us what someone recorded and what later custodians preserved; they cannot automatically tell us how representative those records were of every voice or practice at the time. A surviving account is evidence of an account, not a census of everything that happened. The missing record may be unknowable, and we should not invent it to complete a pattern. We can, however, remain conscious of the selection that brought the surviving record to us.

The temptation in both settings is to mistake a well-lit display for the full room. A celebrated experiment is easy to describe. A practice preserved in a book is easy to quote. A polished theory is easy to teach. What resists easy description—the unreported comparison, the alternative interpretation, the question that produced an awkward answer—may be exactly what would keep our judgment from becoming too neat.

This does not require permanent suspicion of everything we read. Suspicion without discrimination is merely another way of refusing to learn. It suggests a more useful habit: when a claim rests on a striking study, ask whether the question and outcome were defined beforehand, whether other attempts are known, and whether the effect has been tested again. When a history feels unusually seamless, ask which competing explanations the author had to rule out. These are not demands for impossible certainty. They are requests to see the edges of the picture.

The great achievement of a library is that it lets knowledge travel beyond the people who first made it. But the existence of a shelf can also give an impression of completeness that no shelf deserves. The absence of a book does not prove that its subject was never studied. The presence of a book does not tell us how many attempts it took to reach its conclusion.

Rosenthal’s drawer was a metaphor for a research problem, not an instruction to admire failure for its own sake. The goal is better judgment. Sometimes the missing study would change the conclusion. Sometimes it would strengthen it. Sometimes it would only tell us that the question is harder than the published story suggests. Each possibility is worth more than a certainty purchased by not looking.

Perhaps that is the obligation a good collection places upon us: not to fill every silence with a theory, but to remember that the silence has a history. Before we ask what the evidence says, there is another question, quieter and less comfortable: what had to happen for this to become the evidence we were allowed to see?

— Aadvik Agastya

KEEP THE QUESTION MOVING

Three ways to continue.