Meaning, Knowledge and Planner Audit
F896 tested whether a natural-language request could travel through learned text-to-meaning, evidence-closed planning, ordered propositions, lexicalization, and final response construction. The structured planner passed its scoped contract, but the complete pipeline did not.
Objective and research question
Objective
Evaluate the complete path from a free-text request to a supported, ordered, and verbalized answer while keeping knowledge selection closed to the evidence supplied by the benchmark.
Research question
Can the system convert an unseen natural-language request into a complete structured meaning, select only supported facts, order them through the planner, and realize a satisfactory final response?
Technical context
The audit deliberately separated stages that can otherwise be confused: text-to-meaning, evidence selection, structural planning, lexical realization, and final decoding.
The planner did not generate text. It selected explicitly supported facts and produced an ordered structural response plan. Its 100% plan-coverage result used an oracle structured query with frozen evidence, not unrestricted natural-language understanding.
The end-to-end measures instead used the actual text-to-meaning output. That distinction is central to the phase verdict.
Method
- The blind set contained 32 cases across eight categories, with four cases per category and no exact repository overlap reported at preflight.
- Planner and evidence-selection metrics were measured on a frozen evidence fixture. End-to-end metrics applied the real text-to-meaning output and the available language path.
- Ablations disabled text-to-meaning, structured memory, the planner, lexical realization, and final decoding separately to identify where performance degraded.
- Cold reload repeated all 32 recorded cases, but restart consistency did not convert the failed end-to-end gate into a pass.
Evidence and benchmarks
| Measure | Result | Scope | Evidence |
|---|---|---|---|
| Structured planner contract | 32/32 | Oracle structured query plus frozen evidence; plan coverage 1.000000. | F896 primary audit |
| Complete text-to-meaning | 3/32 | Actual natural-language input; meaning_complete 0.093750. | F896 primary audit |
| End-to-end plans | 2/28 | Actual text-to-meaning followed by structural planning on the applicable non-defer cases. | F896 primary audit |
| End-to-end responses | 2/28 | Recorded end_to_end_response_success 0.071429; this is not 0%. | F896 primary audit |
| Oracle-plan F711 success | 14/28 | F896 language metric on the 28 supported, language-applicable oracle-plan cases. | F896 primary audit |
| Cold-reload match | 32/32 | Meaning completeness, selected evidence identifiers, and planner acceptance matched; final answers were not revalidated. | F896 primary audit |
Results
The structured, evidence-closed planner achieved full plan coverage and logical order on the oracle-query fixture. Knowledge precision and recall were also 1.000000 within that supplied frozen evidence.
Those values do not mean the system understood all 32 natural-language requests or already contained the benchmark’s synthetic facts. The primary report explicitly limits the knowledge metrics to selection from the supplied evidence fixture.
Only 3 of 32 requests produced a complete meaning. On the 28 applicable non-defer cases, end-to-end plan coverage and end-to-end response success were both 2/28.
The phase gate remained false, no operational checkpoint was promoted, and the terminal decision was insufficient.
Recorded source outcome: F896_PLANNER_CAUSAL_END_TO_END_INSUFFICIENT. The shorter verdict badge above is a conservative public label, not a replacement for the source decision.
Failure analysis
- Text-to-meaning was the first recorded divergence: focus or relation handling was incomplete or unreliable on the first case.
- The language path could realize only part of the oracle-structured set. The first F711 divergence was a requested surface that was not available.
- The report records that final decoding could lose a proposition even when the structural plan was complete.
- The F897 follow-up later isolated an additional transport defect: reordered facts could remain paired with contracts from their previous positions. That later diagnosis must not be retroactively described as an F896 success.
- The F897 baseline also showed that none of the 28 frozen F896 meanings had complete surface coverage before the follow-up repairs.
What the phase demonstrated
- A structural planner can select supplied evidence without adding unsupported propositions in this fixture.
- The planner preserved the required logical order under its structured-input contract.
- Planner ablation degraded the recorded ordering result, supporting a causal role within this benchmark.
- The phase successfully separated upstream understanding, evidence selection, planning, lexicalization, and decoding failures.
What the phase did not demonstrate
- Successful end-to-end natural-language answering.
- General or unrestricted text understanding.
- Universal multi-fact reasoning or answer construction.
- That the frozen benchmark facts were already internal knowledge.
- A production-ready or promoted language path.
Known limitations
- The blind set contained 32 synthetic cases across eight defined categories; it was not an open-domain population estimate.
- Planner metrics used an oracle structured query and supplied evidence, while end-to-end metrics used actual text-to-meaning output.
- The later 0/28 surface baseline comes from the F897 follow-up and measures a narrower surface-completeness gate than the F896 report’s 2/28 end-to-end response metric.
- Cold-reload consistency shows reproducibility of this recorded state, not correctness or generality.
What changed afterward
F897 repaired fact-contract identity transport and extended noun-phrase composition. Its frozen 28-case surface result improved to 14/28, but its overall gate remained closed.
F898 then addressed the remaining predicate-realization gap on the same frozen meanings and passed its own narrower evidence package. That later pass does not change F896’s insufficient verdict.
Public evidence and source figures
No original source figure is present in the current F896 public evidence package. The metric chart is an editorial HTML rendering of the verified table values, not a new benchmark or source figure.
F896 primary audit
Blind-set definition, scoped metrics, ablations, divergences, cold reload, and terminal decision.
F897 follow-up baseline
Later reproduction of the 0/28 complete-surface baseline and the identified fact-contract transport defect.
Research topics
Word Model · Word Memory · Speech · Memory