Experiments / Negative results
Failed Experiments: Why Negative Results Matter
A completed update is not enough. F105 and F575 show how negative findings inform answer evaluation and the next research decision.
What happens when an experiment does not deliver the hoped-for result? In AI research, a failed test can still provide important information.
Elena AILL documents these findings too. In F105, an update was completed successfully, but the answers that followed did not pass the specified evaluation.
F575 presents another case: an experimental candidate was compared with a reference version and performed worse on several checks.
Together, these experiments show why completing a modification does not necessarily improve the system’s behaviour.
F105: a completed change, an unresolved problem
The phase starts from families of errors observed in the previous evaluation and applies a local update. The operation report records PASS: the intended intervention was completed. The subsequent answer evaluation records FAIL.
The two outcomes measure different things. The first confirms that the update was carried out correctly. The second checks whether the answers pass the specified evaluation. In F105, that second check returned FAIL.
Sixty answers judged sensible
In the F105 retest, 60 of 99 answers were judged sensible under the experiment’s criteria. The result was not sufficient to pass evaluation.
The ninety-nine cases belong to the retest. Other checks have their own samples; treating them as a single before-and-after comparison would change what the evidence means. The useful finding is what this evaluation observes, together with its FAIL outcome.
F105 · An evaluation remains FAIL
The retest records 60/99 responses judged sensible. The rest of the bar completes the ninety-nine cases without introducing a new classification of errors. The update operation is PASS; the subsequent evaluation is FAIL. This is not a measure of Elena’s general ability.
Source: F105 · Blind Failure Learning ↗F575: the candidate that trails the reference
Across F575’s 30 comparisons, the candidate was better than the reference in just one case. It was worse in 13 and tied in the remaining 16.
The number of checks passed also differed: 16/30 for the candidate, against 27/30 for the reference.
Relative comparisons and passed checks are two different measures on the same thirty cases. Even an automatically accepted answer can retain defects. The candidate remains research-only and unpromoted.
F575 · Relative outcomes and passing cases
The panels describe the same thirty cases using different measurements; they do not form a sample of sixty observations. Research-only status and the absence of promotion remain part of the finding.
Thirty comparisons: 1 better, 13 worse and 16 ties. Separately, passing cases: candidate 16/30 and reference 27/30.
Source: F575 ↗Understand the error before changing more
Understanding why an answer fails means locating the error. It might occur in interpreting the question, selecting information or constructing the sentence. The F105 results reported here do not establish which of these causes is responsible.
That phase does establish a clear point: completing the update was not enough to pass evaluation. The retest keeps the intervention separate from its effect and identifies the need for further investigation.
The connection with F896
In F896, the gap appears between components: the planner passes its check with structured information, while the path from a natural-language request to an answer remains insufficient. Following the full process exposes the limitation behind a positive local result.
The two phases neither establish the same defect nor produce statistics to add together. They do offer a shared reason to look beyond a successful operation or component: what has changed for the final answer?
Keeping a point of comparison
Documenting a failed experiment records what was tried, what results it produced and which problems remained.
When a new modification is introduced, that record helps determine whether behaviour has actually improved.
Public documentation can make objectives, general conditions, authorised measures and outcomes understandable while keeping proprietary details confidential. Reading a test requires knowing what the evidence supports, not having the procedures to reconstruct the system.
A FAIL that remains useful
F105 and F575 show two different difficulties in Elena AILL’s research. In the first, an update was completed but the answers were still insufficient to pass evaluation. In the second, an experimental candidate performed worse than the reference version.
Both findings are openly documented in the project’s scientific record.
Research advances this way too: testing what works, recognising what does not and using the findings to guide the next experiments.
