Results and corrections
What we have learned
Here we record how findings have changed our own methods and judgements.
2 September 2026
What our forecasting test found
On 2 September 2026, we completed a historical comparison of our structured forecasting method with simpler approaches. Twenty cases produced 180 valid forecasts. Case outcomes were assessed separately, without access to those forecasts. Twelve cases could be scored; eight remained unresolved and were excluded from the result.
On the scored subset, our method performed worse than all three comparators: a contemporaneous reference forecast, the model without our added method, and the model with a general checklist. Related cases were grouped and weighted by group under the predeclared evaluation procedure.
The table shows differences in two error measures. Positive values mean our method performed worse. Figures are rounded.
| Comparator | Brier difference | Logarithmic error difference |
|---|---|---|
| Contemporaneous reference forecast | +0.20395 | +0.49101 |
| Model without added method | +0.05074 | +0.10888 |
| General checklist | +0.03851 | +0.07102 |
Eight open cases limit the overall conclusion. On the scored cases, the added method did clearly worse. A successor must earn its result on new development and evaluation data; this test stays unchanged as the benchmark.
16 August 2026
Making human contributions visible
Our original view of the system credited human planning, selection and rescue too generously to AI. We now distinguish the performance of artificial components from that of the whole team. That changes what an experiment can establish about autonomy.