Reasoning / Evaluation
Artificial Reasoning: What Do the Tests Actually Demonstrate?
From organised facts to requests that need interpretation: F896 and F885 show why test conditions matter when assessing artificial reasoning.
How can we tell whether an AI system can reason? One approach is to give it different problems and examine how it reaches an answer.
Elena AILL investigates this question through experiments that separate understanding a request from organising information and building a plan.
F896 shows why the distinction matters: the planner passes every check with already organised information, while the complete path from question to answer encounters greater difficulty.
A prepared problem and a problem to interpret
In F896’s structured check, the planner receives the problem and evidence already represented. It passes 32/32 checks: the experiment measures planning after the request has been interpreted upstream.
Starting from text requests, meaning is complete in 3/32 cases. End-to-end answers succeed in 2/28 applicable cases, excluding those that call for deferring an answer. The contrast helps trace where the path loses effectiveness.
What each fraction counts
The thirty-two cases concern interpretation and the structured checks. Answer evaluation uses the twenty-eight applicable cases. The results do not all come from the same test, so each value needs to be read alongside the number of cases actually examined.
A case may pass through several stages and be checked at each one. Its successes cannot be added into an overall reasoning total. The chart keeps the measures separate to show the path without turning it into a single score.
F896 · The component and the complete path
Structured planner 32/32; complete interpretation 3/32; end-to-end responses 2/28. Each panel has its own denominator. The structured control does not measure natural-language understanding, and the overall F896 gate remains unmet.
Source: F896 · Meaning, Knowledge and Planner Audit ↗Where the planner makes a difference
The structured test checks selection of supplied evidence and adherence to order. Another check examined what happened when the planner was removed. Ordering worsened in that test, showing that the component contributed to the observed result.
The facts are supplied by the test, however; this does not establish that the system had already learned them internally. Nor does the planner write the final sentence. Distinguishing these roles shows what worked and what still needs to reach the answer.
F885: three conditions compared
What changes when Elena AILL’s configuration is altered? F885 compared three conditions: the configuration specified by the test, a version without memory and one with empty memory.
It considered two separate measures: the overall result, labelled Overall, and the result for reasoning, labelled Reasoning.
In the specified configuration, the values were 0.80 and 0.833 respectively. In both other conditions, they fell to 0.20 and 0.083.
The comparison shows that configuration affects the observed results. Because the changes also involve learned state, the test cannot attribute the difference to a single memory component. The phase records PASS and the decision VALIDATED within its scope.
F885 · Configuration dependence
Overall retains two decimal places from the public figure; Reasoning retains three from the summary. The ablations do not isolate each module or establish superiority over humans or LLMs. This experiment remains separate from F896’s structured control and complete pipeline.
Valid configuration: Overall 0.80, Reasoning 0.833. No memory: 0.20 and 0.083. Empty memory: 0.20 and 0.083. Common 0–1 scale.
Source: F885 ↗Following an error to the end
An orderly plan can address the wrong problem if the initial interpretation is wrong. A correct plan can instead lose information in the final sentence. Knowing where the process breaks down matters when deciding what to change.
F896 keeps these steps distinct and retains the overall verdict: gate unmet, outcome INSUFFICIENT, no promotion. The planner’s success describes an important part of the experiment, not a replacement for end-to-end evaluation.
How one test informs the next
F896 helps identify stages that need further testing. F898 addresses a different problem: making linguistic forms available for meanings that are already structured, rather than establishing general understanding of requests.
The improvement in F898 does not change F896’s historical outcome. An unpromoted candidate also remains distinct from the current runtime.
Look at the task before the score
The results of a reasoning test also depend on the conditions under which it is run.
In F896, Elena AILL’s planner passed all 32 checks when the problem was already structured. Starting with natural-language requests, the complete path passed 2 of the 28 applicable cases.
Research continues on the connection between interpretation, planning and answering. The aim is to improve the complete path, not just its individual components.
