Secondary domain validation. Demonstrates the same bounded retrieval architecture applies to conversational memory, not just AV telemetry. Not a primary TeleMemetry claim.
A 1000-question LongMemEval V1 run (500 S + 500 M, oracle excluded) is currently executing on RunPod using the AIFS bounded retrieval adapter. Official GPT-4o judge scoring will run on completion.
This page will be updated with official scorer results when the run completes. All numbers here are preliminary and not yet scored by the official LME judge.
Why LongMemEval
TeleMemetry's primary claim domain is high-frequency operational telemetry: AV sensor streams, industrial sensor logs, robotics operational archives. nuPlan is the primary proof.
LongMemEval V1 is a conversational memory benchmark: 1000 questions across single-session and multi-session chat histories. Running LME demonstrates that the same bounded retrieval architecture works across a structurally different input type: episodic conversational records rather than deterministic sensor frames.
If the architecture generalizes here, it is not a brittle point solution. That is the architectural claim, not a recall claim on conversational memory specifically.
Dataset and run configuration
Dataset: LongMemEval V1, publicly available. 500 S (single-session) questions and 500 M (multi-session) questions. S and M share the same 500 question_ids by dataset design. Oracle split excluded. Total: 1000 questions.
Deduplication: Dedup key is scoped to split:question_id to correctly handle the shared question_id space between S and M splits.
Hardware: RunPod GPU instance. Model: qwen2.5:7b via Ollama.
Scoring: Official LME scorer (evaluate_qa.py gpt-4o) with GPT-4o as judge. JSONL output format: question_id + hypothesis fields. The only legitimate score is the official scorer output.
Known failure analysis
A prior local diagnostic run identified the dominant failure mode: wrong-session selection. The retrieval layer retrieves the correct type of information but from the wrong session context. Replay recall is 100% everywhere (the record exists in the archive); the failure is at the retrieval step, not the memory step.
This failure mode is distinct from forgetting. The information is in the archive. The retrieval selector is pulling from the wrong session. Keyword rescue helps in approximately 30-50% of wrong-session misses.
Worst blocks from diagnostic: 901-950 (42% bounded recall), 826-850 (40%). These are retrieval accuracy figures from the diagnostic run only, not from the current scored run.
Claim boundary for this domain
Run log
2026-06-27: Local diagnostic run (6 questions). Retrieval failure identified: hit_sessions empty due to pipeline bug.
2026-07-03: RunPod run started. Adapter confirmed correct after 3 patches: all_no_oracle split handling, dedup key scoped to split:question_id, preflight uniqueIds scoped to split.
2026-07-10: Current run (PID 9233) in progress. Preflight passed: 1000 total (500 S + 500 M). Official judge scoring pending.
This page will be updated with the official score. Check back after run completion.