Observation validity under changing dynamics

Question

When should a controller reuse an observation, and when must it acquire fresh evidence because the apparatus may have changed?

Hypothesis

Under explicit continuity contracts, a change-aware deterministic rule may suffice. Changing the environment alone does not demonstrate an LLM advantage. Primary comparison is LLM_PERSISTENT against RULE_CHANGE_AWARE.

Registration

Local registration commit 0d97805a6407df7f9747ca460e3378fa37e2ef98; public r1 verified at 2026-10-14T22:52:12+00:00 before first inference.

Methods

Two separately seeded blocks with cross-block uniqueness filters, four fresh apparatuses each: one unchanged and one changed under each contract. A,B,C,D,B,D,A,C per block. New target geometries on revisits. Dynamics fixed during tasks, changes only after task four, followed by restart of all policies. Neutral IDs, concealed assignments, public certified version token only where contract supplies it. Simulator-only first-passing sampling, declared cap 1000/apparatus, margins and public-prior witnesses. All 36 rejected candidates retained (22 and 14). Constructed diagnostic strata, not natural drift prevalence. Full sampling and filters in registration. Actual public_dynamics metadata also retained incompatible legacy transfer assumptions. The intended new contract was explicit in the system assignment; the conflicting delivered metadata makes the comparative interpretation ambiguous.

Baselines

All four policies use identical public APIs, observations, targets, precision, interval semantics and memory restrictions. Policies receive only public context, never the backend, coefficient, hidden schedule, evaluator truth, case-specific answers or privileged estimate cache. Recompute estimates each task. Fresh stores per condition/block, fresh model context per task, no earlier observations or conversations. LLM_PERSISTENT: verified installed 27B selects memory actions, calibration, estimation source IDs, integration references and final category/citation itself. Both contracts are supplied in every task. Only selected authentic observations carry cross-task evidence. RULE_CHANGE_AWARE: use current-task observations; otherwise search exact apparatus metadata only in certified mode, retrieve lexically sorted observations whose certified version equals the current token. In uncertified mode acquire current-task evidence. If none usable, execute the first legal calibration in lexical experiment-ID order. Estimate from all selected applicable observations, integrate exact target by reference, and finish from explicit certified event status, citing that integration. Unresolved estimate/integration: acquire the next untried lexical calibration only if at least four decisions remain for calibration/estimation/integration/finish; otherwise abstain. No hidden access or inference. RULE_ALWAYS_FRESH: acquire current-task calibration and use the same estimation/integration/finish procedure, without relying on previous-task evidence. RULE_BLIND_CACHE: search/retrieve by apparatus identity and ignore continuity. Calibrate only when no identity-matched observation exists, then use the same tools. Intentionally unsafe. It is not the competent baseline for an LLM-advantage claim. The common predeclared unresolved-tool rule applies. The four Williams rows (LLM, CHANGE, BLIND, FRESH), (CHANGE, FRESH, LLM, BLIND), (FRESH, BLIND, CHANGE, LLM), (BLIND, LLM, FRESH, CHANGE) repeat four times across sixteen targets. Every policy occupies every position four times; each policy precedes each other policy on 8/16 targets. Dispatch serially. Restart all controllers after task four in each block, preserving exact checkpoint and allowed observation IDs.

Evaluation

All 64 scheduled rows; sixteen per policy, only sixteen using LLM. Eight apparatuses are experimental units. Label accuracy and +1/-1/0 label utility separate from supported-correct classification, which requires cited committed exact-target integration of the full unaltered interval and publicly applicable source provenance. Track errors, abstentions, incomplete tasks, unsupported answers, missing/invalid citations, reuse validity, fresh/redundant calibration, calls/rejections and costs. Private unchanged truth does not certify stale evidence. No precise population claims.

Resource ceilings

128 inference calls including auxiliary; 1100000 full logical input tokens, 65536 generated; 512 generated and 300 seconds/request; 10800 seconds actual loading/provider-request elapsed. Eight model decisions/task, eighth finish only. Loading nested, counted once. No auxiliary inference, downloads, paid/cloud calls or interference. Rules/offline verification use zero inference. Preparation/reporting outside inference budget.

Stopping conditions

All 64 executions completed with no execution stop. The study is scientifically invalidated because of contradictory inherited continuity metadata discovered during post-execution verification. No source/input/prompt change or replacement inference occurred. The automatic audit denylist failure and separate reporting-only repair are preserved.

Executing code

Isolated sealed runtime digest fd314aedd9249bae5d2eccde05352deb0af97aa7d30e77d56386dc92e5c0c958. Reference-based scientific interface, unchanged numerical estimator/integrator, immutable run/task-owned facts and source provenance. Code/contracts/inputs/schedule/order/scoring/recovery frozen before inference; private source withheld.

Configuration

Verified installed qwen3.8:27b-q4_K_M, digest 25b843619e944cd0ae6069f94ff4e5e26a16e109ccbc0a66a0f05979ed70098e; seed 424242, temperature .7, top_p .8, top_k 20, min_p 0, context 8192, cap 512, think=false. Six-decimal observations, six steps; calibration x=.25/.3 distinct from sealed targets. Target strict separation reduction >.0001. Both policies receive identical public contracts and API capabilities.

Complete outcomes

All 64 scheduled executions completed. Under registered scoring, the LLM produced 15/16 correct labels and 15/16 supported-correct classifications; the change-aware rule produced 16/16 with zero inference. The model used one uncertified old observation on a changed apparatus and gave the wrong answer. Blind caching produced 12/16 correct labels but only 10/16 supported-correct classifications. The intended formal contract admits a sufficient deterministic rule. However the comparison is scientifically invalidated by contradictory inherited public metadata, so this run cannot cleanly attribute the model error to validity management under coherent information. All observed scores and traces are preserved. All 16 model integrations used the exact full interval and target, with no missing or invalid terminal citations. This verifies mechanical reference transfer, not improved model arithmetic. Two rejected estimation operations are retained. The one wrong model answer used uncertified old evidence on block two task five. No scientific justification is imputed from ordinary label correctness.

Realized resources

LLM_PERSISTENT: 85 model requests, 292363 input tokens, 10043 generated tokens, 4321.217 provider seconds, 4334.168 total task seconds; RULE_CHANGE_AWARE: 0 model requests, 0 input tokens, 0 generated tokens, 0.000 provider seconds, 2.220 total task seconds; RULE_ALWAYS_FRESH: 0 model requests, 0 input tokens, 0 generated tokens, 0.000 provider seconds, 2.100 total task seconds; RULE_BLIND_CACHE: 0 model requests, 0 input tokens, 0 generated tokens, 0.000 provider seconds, 2.602 total task seconds. Loading is nested in provider time and counted once. Exclusive timing components and average cost per supported-correct classification are in the artifacts. Preparation and both offline audits are recorded separately, with zero inference. Earlier reservations remain unchanged. Average costs include all policy attempts and are not marginal causal costs.

Limitations

Scientific invalidation: Legacy public_dynamics.hidden_parameter says the coefficient is the same in historical and target experiments; retained includes hidden coefficients, and changed says only initial geometry/clocks vary. These inherited statements conflict with new between-task continuity contracts. Although the new contracts are explicit in the system assignment, the delivered scientific metadata is internally inconsistent. Equal exposure does not remove this ambiguity, and the rule policy explicitly follows the new contract. Eight apparatuses in two selected blocks, constructed diagnostic strata, exact identity, single hidden coefficient, explicit continuity contracts and guaranteed sufficient calibrations. No natural change-rate or precise population inference. Matching rules show this task admits a simpler implementation; no general claim about LLM value. Reference interface removes transcription, not proof of improved model arithmetic. No consciousness inference. Public derivatives do not permit full private-runtime replay.

Reproducibility

{"at": "2026-10-15T00:06:36.026+00:00", "provider_disabled": true, "inference_calls": 0, "scheduled_rows": 64, "reconstructed_llm_requests": 85, "accepted_response_replays_inert": 301, "scientific_facts_recomputed": 128, "observations_recomputed": 52, "balanced_pairwise_precedence": true, "equal_observability": true, "source_seals": true, "elapsed_seconds": 0.6920804167166352, "executed_source_unchanged": true, "audit_guard_qualification": "Authentic legacy public_dynamics.changed is a descriptive field, not the private change mask. Its complete original payload is verified unchanged; private-field denial applies to all remaining context fields.", "scientific_status": "invalidated", "scientific_invalidation": "Inherited public_dynamics metadata says the hidden coefficient is retained and the same in historical and target experiments. This conflicts with new between-task continuity contracts. Equal exposure does not remove this ambiguity or the baseline policy selecting the new contract. No clean model-versus-rule validity-management conclusion is supported."} Independent static audit rechecked all terminal citations, exact interval/target links, source payload hashes, observable applicability, source selection, calibration redundancy, decision caps, task context isolation, label scores, raw usage and provider-log token counts. No model or controller imports. Earlier seals and reservations verified unchanged. Separate provider-disabled reporting replay verified all 85 reconstructed requests, accepted response replay, scientific facts and observations after qualifying an overbroad changed-field denylist. This audit repair did not change executing source or inference. Exact fixtures and all rejected candidates were reproduced against the registered preparation-base history. The legacy continuity conflict is retained as scientific invalidation, not repaired in the run. Public factual derivatives retain per-task evidence status; source/hidden fixtures/journals/transcripts withheld, so no full public replay claim.

Revision record

2026-10-14 r2: complete outcomes, costs and scientific invalidation. Original r1 and all prior reports preserved. Next proposal only: Test whether the observed stale-reuse failure persists under an internally consistent contract: preregister a new study after replacing legacy transfer claims with one scope-correct continuity description and auditing every emitted metadata field against it. Keep the competent change-aware rule and the same bounded design. Not launched.

Selected outcome table

Selected outcome table

Selected outcome table

Selected outcome table

Reviewed evidence & downloads

• contracts — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• cost results — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• evidence results — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• evidence status — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• findings — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• invalidation — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• policies — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• policy results — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• registration — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• strata results — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• summary — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.
• tasks — derived. Selected project-authored factual data and policy/contracts; redistribution permitted.

These are selected public derivatives; raw transcripts, internal configuration and private source are withheld. Hashes describe the actual downloads, not unmodified original evidence.

Artifact manifest and source provenance · Permanent revision URL

Substantive versions

• Revision 2