The name is not the spec
Pick an open-weight model from a hosted API and the listing gives you a name, a price and a speed. What it does not give you is how much of the model's accuracy survives that provider's serving stack. Quantization, custom kernels, output caps, tool-call parsing and plain bugs all sit between the weights and your application, and until now buyers had no standard way to price that gap.
Artificial Analysis built one. Its Endpoint Accuracy Index runs a fixed evaluation suite against each provider endpoint and a self-hosted reference deployment using the official weights at the lab's recommended precision. The reference configuration and serving commands are published, giving buyers a common comparison point. An endpoint scores 100% when it matches that reference, and each result carries a 95% confidence interval so a reader can tell a real deficit from sampling noise.
Version 1.1, published on November 6, expands coverage to six models and tightens the long-context test: AA-LCR-25 now runs 25 repeats instead of 10, which narrowed the average confidence interval from ±4.7 points to ±2.6, and two answer keys were corrected. Across the more than 100 endpoints now covered, about one in four scored numerically below its reference, according to AlphaSignal's summary of the release.
Where the points go missing
The losses are rarely mysterious. On GLM-5.3 Flash, two endpoints capped output near 32,000 tokens and cut off about half of their HLE-250 responses before a final answer appeared. Their raw accuracy on that test was 22% and 29%, against 55% for the reference. Two other endpoints produced roughly 30% fewer reasoning tokens without any recorded truncation, and scored 48%, seven points below the reference.
Tool calling fails in a different way. On Meta's Muse Spark Open, three of five endpoints tested at the highest reasoning setting scored between 99.5% and 99.9% of the reference. The other two encoded tool arguments declared as strings, such as booking IDs, as numbers. Their raw tool-calling scores fell to 50% and 51%, against 59% for the reference, while their reasoning and long-context scores stayed within two points of it. A team that never calls tools would never see the defect. A team building an agent would hit it on the first day.
GLM-5.3 shows the full range: 26 endpoints from 67.2% to 99.3% of the reference, with the largest deficits again traced to output caps and reasoning loops that truncate responses. One result cuts against the usual assumption. Five of the six endpoints serving GLM-5.3 at FP4 precision were statistically consistent with the FP8 reference, so lower-precision weights alone do not explain most of the observed differences in this sample. The individual failures point instead to serving configuration, including output limits, reasoning behavior and tool-call parsing.
The contrast case
DeepSeek's provider results are the tightest in the index. All 22 endpoints were statistically consistent with the reference, spanning 97.5% to 101.4%. That matters because it suggests the test does not automatically produce large provider differences: the same methodology can also show near-parity when providers serve a model carefully.
Price does not settle the question either. Separately from the Endpoint Accuracy Index scores, AlphaSignal's summary reports blended costs of $0.41 to $1.81 per Intelligence Index task for GLM-5.3 Flash endpoints, a 4.4-fold range, with completion times from 5.5 minutes to 51.5 minutes. These are cost and timing comparisons on the Intelligence Index task basis, not additional accuracy measurements from the Endpoint Accuracy Index. An endpoint scoring 79% or 83% of the reference can still look attractive on a price or speed table. The accuracy column is the one that was missing.
What to do with this
The index covers a fixed suite, and its own methodology page is clear that results are dated snapshots, not live monitoring; a provider can fix a parser or raise a cap the week after it is measured. Treat a low score as a prompt to check, not a permanent verdict. The practical order is: shortlist on price, latency and region, then check the accuracy score and its confidence interval for the exact endpoint, then read the per-test breakdown that matches your workload. Tool calling for agents, hard reasoning for analysis, long-context recall for document work. They fail independently, as the Muse Spark Open result shows.
Sources
- [1] Artificial Analysis, Endpoint Accuracy Index v1.1 methodology (the fixed suite: BFCL v4-500 with 3 repeats, HLE-250 with 10 repeats and AA-LCR-25 with 25 repeats in v1.1, up from 10, weighted equally; each endpoint scored as a percentage of a self-hosted reference deployment running the official weights at the lab's recommended precision, with reference configurations and serving commands published; results are dated snapshots, not live monitoring; v1.1 moves AA-LCR-25 to dataset version 1.1.1, correcting the answer keys for two questions)Read source
- [2] Artificial Analysis, "Launching the Endpoint Accuracy Index: Same Model, Different Accuracy", August 4, 2026 (the v1.0 launch of the index; framing and the original GLM-5.2 / gpt-oss-120b findings)Read source
- [3] AlphaSignal, "Artificial Analysis Finds 1 in 4 AI Endpoints Quietly Underperform Their Advertised Models", November 6, 2026 (the summary that about one in four of the more than 100 serverless endpoints scored numerically below its reference deployment; GLM-5.3 Flash's 22 scored endpoints from 24 providers, including the two output-capped endpoints at 22% and 29% raw HLE-250 accuracy against 55% for the reference and the two endpoints at 48% with roughly 30% fewer reasoning tokens; the Muse Spark Open tool-argument findings; GLM-5.3's 26 endpoints spanning 67.2% to 99.3%, with five of six FP4 endpoints statistically consistent with the FP8 reference; DeepSeek V4.1 Flash 1017's 22 endpoints spanning 97.5% to 101.4%, all statistically consistent with the reference; the average 95% confidence interval narrowing from ±4.7 points to ±2.6 points; blended costs of $0.41 to $1.81 per Intelligence Index task and completion times of 5.5 to 51.5 minutes for GLM-5.3 Flash endpoints)Read source
- [4] Artificial Analysis, GLM-5.3 Flash provider results (24 API providers tracked, with per-endpoint accuracy, price and time-per-task figures)Read source