How TeleMemetry relates to MLPerf, what MLPerf measures, what it does not measure, and where TeleMemetry sits in that space. Not a competing benchmark.
What MLPerf measures
MLPerf is an industry-standard benchmark suite measuring machine learning system performance: inference throughput, inference latency, and training throughput. It covers image classification, object detection, natural language processing, recommendation, and speech recognition workloads.
MLPerf answers: how fast can this system run inference? How many inferences per second? At what latency percentile?
MLPerf does not answer: can this system maintain accurate memory of operational history over millions of records? Can it retrieve a specific past event from a growing archive with bounded token cost?
Where TeleMemetry fits
MLPerf measures the inference layer: how fast a model processes a given input. TeleMemetry measures the memory layer: what input gets constructed for the model when the operational archive is too large to pass directly.
A system could score well on MLPerf inference benchmarks and still be unable to handle a 2M-record operational archive without full context replay. MLPerf does not characterize this problem because it does not model growing archives. TeleMemetry does.
These are complementary measurements of different layers in an AI system stack.
MLPerf submission requires running on specific hardware configurations in controlled environments, with results validated by MLCommons. TeleMemetry has not submitted to MLPerf.
More importantly, MLPerf workloads are not designed to measure memory recall over growing archives. A TeleMemetry MLPerf submission would require a new workload category that does not currently exist: sustained operational memory with factual retrieval recall as the metric.
TeleMemetry throughput context
These are TeleMemetry benchmark numbers, not MLPerf numbers. They are not comparable to MLPerf submissions because the workload definition, hardware certification, and metric definitions are different.
What a TeleMemetry MLPerf category would look like
A TeleMemetry workload in an MLPerf-style framework would define:
Archive depth: N records (e.g., 1M, 10M, 100M).
Probe frequency: One factual probe per K records.
Token budget: Maximum context tokens per probe (fixed).
Metric: Recall % at that token budget, across that archive depth.
Scaling law: How does recall change as N grows, for fixed budget?
The fundamental measurement would be: what is the recall@budget curve as archive depth scales? Systems that maintain recall at depth with a fixed budget would score well. Systems that require growing context windows would not be viable at scale.
This is speculative. No such MLPerf category exists. TeleMemetry has not submitted to MLPerf. This section is for technical buyers evaluating how to contextualize TeleMemetry relative to existing benchmark frameworks.