How the benchmark works, how recall is scored, and how to run it yourself.
TeleMemetry measures whether a bounded retrieval system can answer factual probe questions over a long operational archive without replaying that archive as model context.
It does not measure model reasoning quality, hallucination rate, answer fluency, or performance on open-ended questions. The evaluation is scoped to factual recall of indexed operational values.
Definitions
One ingested telemetry frame from the nuPlan Mini AV dataset. There is no LLM call during ingestion. The engine reads the record, adds it to the archive, and continues. A 1M-record run ingests 1,000,000 records.
A factual question about a specific past record, issued at a fixed interval during a run. The system constructs a bounded retrieval set, calls the model with that set and the question, and scores the answer. Probe interval is specified per run configuration (typically every 1,000 to 10,000 records).
The evidence set constructed for a single probe. This set has a known maximum token count, configurable per run. It does not grow with archive depth. The system selects the relevant records from the archive using the retrieval layer and constructs the set within the token budget.
Recall % = (probes passed / probes issued) × 100
A probe passes when the model answer cites or matches a specific measurable value from the source record: speed, heading, object class, timestamp, position, or a similar indexed field. The pass/fail threshold is documented per benchmark category.
For nuPlan benchmarks, a pass requires the answer to cite or match the specific measurable value from the source record. Partial matches and paraphrased values are evaluated per the scoring methodology in the run artifact.
Dataset
nuPlan is a large-scale AV planning dataset from Motional / nuTonomy. The Mini subset used in TeleMemetry benchmarks contains real nuPlan telemetry replayed from stored files as a streaming feed. It is not live vehicle data, not an NVIDIA DRIVE integration, and not a production safety validation.
Data lane: REAL NUPLAN SUMMARY. No raw telemetry frames are included in public artifacts. Numbers in public result cards come from the run summary JSON and were not adjusted or normalized after the fact.
Data integrity
Each certified run artifact is published as a compressed archive (.tgz) with a SHA256 checksum file. The Integrity Root is computed over the per-probe scoring log and published alongside the SHA256. These allow independent verification that the artifact has not been modified after the run completed.
Verification steps:
1. Download artifact and .sha256 file
2. sha256sum -c artifact.tgz.sha256
3. Extract and compare Integrity Root against published value
4. Inspect per-probe log for individual question pass/fail records
Reproduction
The public benchmark kit contains: the verification harness, the public nuPlan subset (200 records), the scoring methodology document, and SHA256 sums.
Running the public smoke test exercises the full scoring path against the 200-record subset with 200 Q&A probes. Expected time: 5-15 minutes on a GPU with 12GB+ VRAM. Expected result: 200/200 probes passed.
Full 1M-record reproduction requires the core engine image, the nuPlan Mini dataset (~50GB), and a GPU with 12GB+ VRAM. Expected time: 7-12 hours depending on GPU throughput.
The core engine binary is not currently open source. Independent reviewers can request engine image access at AIFSOfficial@proton.me. Review requests are evaluated for technical legitimacy and responded to within 5 business days.
Claim boundary
This benchmark proves one thing: the bounded retrieval system produced the reported recall rate across the reported number of probes, with the reported number of integrity failures, in the reported run configuration.
It does not prove model superiority, universal memory, industry-wide performance improvement, production deployment readiness, or performance on datasets not in the registry. See claim.html for the full scope boundary.