Three numbers, one announcement
Start with the discrepancy, because it decides how much of the rest to believe. The technical paper, in its abstract, results section and conclusion, claims up to 2.6 times higher tokens per second than the best open-source inference engine configurations on four frontier models, based on InferenceX baselines dated October 27, 2026. The Compiler blog post's body says the same. The Compiler post's own page description, the text search engines and link previews will carry, says up to 3.7× higher throughput. That figure is not in the paper's language-model results at all. It does appear once in a different table: Aurora 1.5, a weather model, scored 3.7× in a set of beyond-language-model experiments where the reported range runs from 1.5× to 6.4×.
Nobody needs to invent a motive. The paper does contain a 3.7× result, but in a separate beyond-language-model table: Aurora 1.5, a weather model, scored 3.7× in experiments where the reported range runs from 1.5× to 6.4×. Whether that result explains the page description is unknown. But a reader comparing inference providers will meet the 3.7 first and the 2.6 only if they read the paper. Treat 2.6 as the claim. Treat the 3.7× figure as unexplained metadata until Together identifies the language-model result it refers to.
The 2.6 itself needs its conditions kept attached. "Up to" is doing work: it is the best setting across four models and five hardware platforms, not the median. The baselines are the best reported open configurations as of October 27, 2026, which in a field moving this fast is already a dated target. For the main comparisons Together reused each leading configuration's parallelism strategy, which is the honest way to isolate the compiler's effect, and the paper says so. The paper also says the optimization process used compute equivalent to twice the replica size per model compilation to support the tiered persistent engine, and that each frontier result was produced within a 24-hour compilation window per setting, using at most two nodes per model. The speed was achieved with substantial optimization compute, and the paper itemizes that cost rather than presenting the result as free performance.
Where the speed actually comes from
The least glamorous number in the announcement is the most convincing one. Testing a candidate optimization inside the tuning loop traditionally means initializing an inference engine from scratch: 93 seconds for initialization, 50 for weight loading, 21 for KV-cache sizing, 93 for JIT warm-up, 193 for autotuning and 10 for API readiness — 467 seconds in total. Together's persistent engine keeps the model loaded, hot-reloads candidate forward passes and reduces the equivalent iteration step to 9.0 seconds: 1.7 seconds for RPC and compiler setup plus 7.3 seconds for CUDA-graph capture. Together says the faster setup enables 252 times more agent experiments than baseline agentic optimization systems. That figure is the company's own comparison, not an independently verified benchmark result. An agent that waits eight minutes per experiment has a working day of a few dozen tries. An agent that waits nine seconds can be wrong productively.
The low-latency records show the same pattern, plus one judgment call worth noticing. On MiniMax M3.1-Flash, Together reports 1,111 tokens per second at batch size 1 on a B200, against 633 for the best published InferenceX result. For that run its cost model overruled the baseline's published parallelism, choosing TP8 over TP4 on the reasoning that decode at batch size 1 is memory-bound. The paper then adds the sentence that keeps the claim honest: running the same baseline image under TP8 performs similarly to its TP4 result. The configuration change alone, in other words, does not produce 1,111. The compiled forward pass does, on Together's own measurement.
The second low-latency result is 1,205 tokens per second on DeepSeek v4.1-Flash using an NVIDIA GB300, also at batch size 1. Together presents it as part of the same Figure 8 low-latency evaluation.
Two further results are narrower than they will be quoted. On Llama 1B, the canonical megakernel workload, the compiler reaches up to 1.7× the hand-tuned megakernels, and Together's Polymorphic post concedes the wider point its own team helped create: after putting megakernels into production, it concluded that kernel fusion is not the right design goal for varied, dynamic workloads. For these beyond-language-model workloads, Together reports speedups of 1.5–6.4× while maintaining bit exactness to the original forward pass, tested on an H100. That correctness claim applies to those tested workloads; it should not be read as a blanket guarantee for every model, workload or hardware platform the compiler might later optimize. On AIFS-ENS 2.0, the Claude Code baseline failed to find a correct solution, while TKCC scored 3.5×. The comparison was against Claude Code using the /goal command and similar context. It shows the value of a purpose-built optimization system, but it is not a general comparison against human engineers or other coding agents.
The Trainium2 results carry the paper's own asterisk. They ran on a synthetic multi-turn dataset with shorter contexts than the benchmark traffic, because the implementation does not yet support the higher concurrencies inside that chip's memory capacity. Trainium2 is therefore not directly comparable with the other hardware results. Its runs used synthetic multi-turn traffic with 70,000-token input sequence lengths and shorter contexts than the AgentX benchmark traffic because the implementation does not yet support the higher concurrencies within Trainium2's available HBM capacity. The Trainium2 figures should not be described as benchmark-traffic results.
What a customer can actually get
As of November 4, Together says Together Compiler is running in production across many of its own top workloads and some popular serverless endpoints. That does not mean external customers can deploy it themselves today. End customers can request access to a limited beta. The self-serve feature described by Together — one-click optimization of a dedicated endpoint — is described as "soon," not generally available, and Together has published no price for the beta. Together also says the system serves trillions of tokens a day across its production fleet.
That gap is the practical story. The pitch of polymorphic inference is that optimization should belong to the deployment, tuned to your model, your traffic and your hardware. For now the deployment it belongs to is Together's. A team serving open models on its own infrastructure gets a paper with a reproducible-looking protocol and no code to run it with. A team already paying Together gets to ask whether the beta reproduces the gains on its traffic, which is the only version of the question that matters, and the one the announcement cannot answer for them.
There is also a quieter admission buried in the results section. Replaying its own production traffic, Together found the leading InferenceX configuration for MiniMax M3.1-Flash hits a capacity ceiling at low request rates, with tail latency degrading 10–40× and 17 percent of requests failing. That is the company's own evidence for its thesis: a benchmark-winning configuration can be the wrong configuration in production. It is a good argument. It also cuts both ways, and readers should let it. The headline performance comparisons were produced by Together under conditions it selected, against baselines dated October 27, 2026, and have not yet been independently replicated. Its production-traffic observations are also Together's own. The method is documented in enough detail to invite checking, but until outside parties reproduce the results, the 2.6× figure should be described as a well-documented vendor claim rather than an independently established result.
Sources
- [1] Together AI — "Polymorphic Inference: self-tuning inference for open models", November 4, 2026 (production deployment claim, limited-beta access, self-serve "soon," megakernel reversal, traffic-shape figures)Read source
- [2] Together AI — "Together Compiler: Continuous Compilation for AI inference", November 4, 2026 (2.6× body claim, 467s vs 9.0s startup table, 1,111 / 1,205 TPS low-latency figures, and the page description's "up to 3.7× higher throughput" wording)Read source
- [3] Together AI Research — "Together Compiler: Continual Compilation for AI Inference", technical paper, November 4, 2026 (abstract and §5 results, §5.2 protocol and October 27, 2026 baseline date, 2× replica optimization compute, Trainium2 synthetic dataset, TP8 note, production-traffic replay finding, verification suite and bit-exactness tables)Read source
- [4] Unite.AI — "Together AI Brings Continuous Compilation to Production Inference", November 4, 2026 (corroborating summary only — its speedup range appears in neither Together document and is not used)Read source