# AI can steer a lab before it can run one.

Claude narrowed 3,500 candidate systems to 20 reports. The human gets the final decision. The AI has already shaped the options.

By Stefan Nilles

Article: https://singularity-intel.ai/analyses/what-a-failed-ai-experiment-proves

## The whole idea in the 1,000 most used words

A computer brain looked through 3,500 possible leads for a study and picked 20 to write about. People read what it wrote, did the work with real things and found something new. That finding is good news. It does not tell us how good the thrown away leads were. People have the last word, but the brain has helped pick what they get to consider.

In another study, computer brains did whole pieces of work. Their own checks found real problems. Yet the work mostly stayed in the same directions. So check what gets started, and what keeps getting time and money. Try checking some thrown away ideas beside the kept ones, with the same time and money. Then see whether a check helps more when people can move money to another plan. Look at the good things found and the time and money lost on wrong turns. Changing your mind alone gets no good mark.

[After xkcd’s Up Goer Five.](https://xkcd.com/1133/)

## Essay

A scientist can approve every experiment and still inherit an agenda selected by AI.

In Anthropic's new biology programme, Claude agents narrowed 3,500 candidate systems to 20 for detailed reports. Human scientists reviewed the work and did the laboratory experiments. The search produced ART, an unusual enzyme system whose function the team is still investigating. [Anthropic's 23 September account](https://www.anthropic.com/news/claude-discovers-novel-enzyme-system)

The discovery gets the headline. The cut from 3,500 to 20 establishes a different kind of scientific influence: shaping the options that receive expert attention. The principal investigator keeps the final say. Most of the candidates have already left the room.

This is why I think the argument about autonomous scientists is missing an important intermediate state. AI can acquire substantial influence over a research programme while still being poor at completing one. Selection deserves its own capability test, well before a machine can replace the person running the lab.

## The shortlist is part of the science

Filtering is essential. Nobody can investigate every plausible lead. A useful agent can open a much larger search space, make neglected connections and spare experts weeks of preliminary work. The relevant comparison is with the laboratory's existing process, complete with its time constraints and blind spots.

But an expert's approval of the survivors answers a different question from whether the right candidates survived. A brilliant discovery validates a finding. Assessing the filter also requires looking outside its shortlist.

[Nathan Benaich made the underlying problem explicit in August](https://press.airstreet.com/p/ai-needs-sciences-search-history): scientific search histories omit abandoned alternatives, and recording a choice does not reveal how the unchosen experiment would have turned out. He proposes sometimes funding the runner-up. This builds on the statistical problem of [selective labels](https://www.cs.cornell.edu/home/kleinber/kdd17-selective.pdf): decisions determine which outcomes become available to learn from.

Anthropic's workflow makes that problem immediate. Its team also feeds lessons from accepted and rejected proposals back into Claude's instructions. Agreement with the scientists can improve while the value of the excluded alternatives remains untested. Learning a lab's taste and improving its science are separate achievements.

## A critic can be right and change nothing

The shortlist decides what gets started. A second decision follows: when the evidence turns against a project, does the work change direction? A study called CRUX puts that decision under pressure.

Peter Kirgis, Sayash Kapoor and colleagues gave agents two unpublished research questions, six days and a $3,000 API budget per run, plus computing resources. Both main papers were rejected by the original researchers. A robustness experiment with another model and scaffold reproduced central failures. [CRUX study](https://arxiv.org/html/2607.27191v2)

The agents could do the engineering. Their AI reviews also identified serious weaknesses. Yet the work largely stayed on the same paths; the response to fundamental criticism was often to narrow claims and add caveats. Both main runs had abandoned their most ambitious targets early. They were capable of rejecting a direction, but poor at judging when rejection or persistence was warranted.

The caveats improved. The research did not improve enough.

Better review can certainly help execution. In [AppliedScientist](https://arxiv.org/html/2609.14738v2), repeated AI reviews guided revisions to code, experiments and manuscripts, with much stronger results on execution problems than on weaknesses in the ideas. The central contribution was deliberately held fixed. That is a useful way to improve a project. Replacing it with a better use of the budget requires a different test.

## Test what enters. Then test what can change.

The distinction I want researchers to measure is between admitting an idea and continuing to fund it. The first shapes what experts get to consider. The second determines whether new evidence can redirect the work. A system can be excellent at either and poor at the other.

Start with the shortlist. Give a human-led process and an AI-assisted process comparable pools of ideas and the same total resources. Include expert attention, experiments and computing in that budget. Then spend part of each budget checking what the process threw away.

Mix randomly sampled rejects with shortlisted ideas, covering different reasons for rejection. Give independent reviewers the same kind of evidence for each, without revealing which group it came from. Count which discarded ideas they restore. Later, count which actually produce useful experimental findings. Being rescued from the bin and being good science are different outcomes.

Now test the decision to continue. Give comparable projects similar criticism and equal budgets. In one process, reviews advise the existing work. In the other, reviews can redirect a reserved share of resources to another approach. Measure useful findings and the cost of wrong turns, including promising work abandoned too early. A system earns no prize merely for changing its mind.

Together, these comparisons locate where AI is helping: better opportunities entering the lab, better decisions after results arrive, or both. They also distinguish a productive shortlist from a process that keeps repairing whichever project happened to get through first. That is more useful to a builder than another undifferentiated claim that an agent can conduct research.

The objection is practical: checking rejects consumes the very attention automation is meant to save, and scientific value can take years to emerge. Put the checking cost inside the comparison, use the same follow-up period, and report what remains unresolved. The filter earns its place if the whole process does better than the alternative. It does not have to become omniscient.

The future of AI science will be shaped by what gets an experiment, not just by who writes the paper.

## Sources & research notes

1. [Claude's biology shortlist](https://www.anthropic.com/news/claude-discovers-novel-enzyme-system)
   23 September 2026; How we work and Claude finds ART

   Anthropic's account of the candidate funnel, human laboratory work and feedback into the agents' instructions. It reports a discovery, not a controlled validation of everything the filter discarded.

2. [Science's missing search history](https://press.airstreet.com/p/ai-needs-sciences-search-history)
   Nathan Benaich, 13 August 2026; A log is not a label, Run the runner-up, Test whether taste transfers

   The closest predecessor on discarded scientific alternatives and testing research taste. Funding a runner-up and retaining rejected branches are prior ideas, not SI inventions.

3. [The selective-labels problem](https://www.cs.cornell.edu/home/kleinber/kdd17-selective.pdf)
   Lakkaraju and colleagues, KDD 2017; pages 1–2

   The statistical problem behind learning only from selected outcomes. Observing the choices a process made does not reveal the outcomes of the alternatives it never tried.

4. [CRUX: agents conducting research](https://arxiv.org/html/2607.27191v2)
   Experimental setup; sections 4.4, 5.3, 6 and 7

   Two main research tasks and the robustness analysis show execution outpacing research judgment. The authors already identify weak backtracking; the essay extends that diagnosis to a separate resource-allocation test.

5. [AppliedScientist](https://arxiv.org/html/2609.14738v2)
   Revision 19 September 2026; abstract, introduction and methodology

   Reviews can improve execution while the central contribution is held fixed. This is useful work, but it tests a different decision from replacing the research direction.

6. [Atria's account of human and AI decisions](https://arxiv.org/html/2609.15818v2)
   Submitted 14 September, revised 17 September 2026; sections 4.1–4.3 and 5

   A team studying its own development process reports AI-generated proposals alongside substantial human final authority and intervention. It helps distinguish option generation, final choice and autonomous agenda-setting; it is not an independent causal productivity experiment.

7. [The wider AI-research feedback debate](https://casp.ac/reports/intelligence-explosion)
   What if automating AI R&D triggers an intelligence explosion?, September 2026; printed pages 3–5 and 7–8 of the linked report

   An adjacent conditional argument about better research agents helping produce better agents, including bottlenecks and oversight proposals. It does not validate a biological shortlist or establish that a lab is autonomous.

### Research notes

The proposed admission comparison needs matched candidate pools and equal total budgets for experts, computation and experiments, including the audit. Randomly sample rejects across different rejection reasons and present them beside selected candidates in the same format to blinded reviewers. Separately count expert-restored candidates and experimentally useful results.

The continuation comparison needs matched projects, comparable criticism and equal budgets, including resources reserved for a pivot. Compare an advisory-review process with one permitted to redirect those resources. Count the benefits and costs of pivots, including premature abandonment, rather than treating every change of course as progress.

Use equal follow-up periods in both comparisons and retain unresolved outcomes. A sample of rejected ideas does not reveal the value of every rejected idea; scientific value may emerge after the test ends. These are proposed comparisons, not reported experimental results.

SI's addition is to separate admission from continuation and evaluate their different opportunity costs. A strong shortlist, good critique and good decisions about changing direction need not travel together. Human final approval can remain genuine while an earlier filter shapes the options.

## Listen

AI narration: https://singularity-intel.ai/audio/what-a-failed-ai-experiment-proves.ogg?v=062d49f97abb50e7d5f30ea401de6ffedc209b29e879eef1d2d776b5217f33ca

Audio SHA-256: 062d49f97abb50e7d5f30ea401de6ffedc209b29e879eef1d2d776b5217f33ca