# dottxt Tool-call Index

> Not all models are served equally by all providers. On a typical open-weight model, the worst provider on OpenRouter breaks tool calls 19× as often as the best.

47 open-weight models, 60 providers, last 7 days; data as of 2026-10-29T00:00:00Z. [Data](data.json) · [For agents](llms.txt)

Rates are OpenRouter's published tool-call error rate per provider endpoint: of the requests that finished with tool calls, the share with at least one invalid tool call (invalid JSON, an unknown function, or arguments that don't match the tool's schema), over the last 7 days, per provider: an OpenRouter provider with its variant (`baseten/fp8`), its regions together. The 19× is the median, across models, of the worst provider's rate over the best's (the best counted as at least 0.1%).

## Every model

Each model's most and least reliable provider, and its recommended OpenRouter routing (`order`, then `ignore`).

| Model | Maker | Providers | Most reliable | Least reliable | Routing | Data |
|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash 1017 | DeepSeek | 32 | Sail Research (fp4) 0.03% | GMICloud (fp8) 6.88% | `sail-research/fp4, primeintellect, ionstream`; ignore `together, novita/fp8, atlas-cloud/fp8, gmicloud/fp8` | [JSON](models/deepseek-v4.1-flash-1017.json) |
| GLM 5.3 Flash | Z.ai | 34 | NextBit (fp4) 0.00% | GMICloud (fp8) 7.78% | `nextbit/fp4, friendli, inference-net/fp4, inference-net/us`; ignore `fireworks, siliconflow/fp8, gmicloud/fp8` | [JSON](models/glm-5.3-flash.json) |
| DeepSeek V4.1 Flash | DeepSeek | 31 | Prime Intellect 0.06% | NovitaAI (fp8) 5.20% | `primeintellect, morph/fp8, relace`; ignore `deepinfra/fp8/flex, novita/fp8` | [JSON](models/deepseek-v4.1-flash.json) |
| Hy4 preview | Tencent | 6 | DeepInfra (fp4) 0.10% | SiliconFlow (fp8) 0.77% | `deepinfra/fp4, novita/fp8, tencent/fp8`; ignore `—` | [JSON](models/hy4-preview.json) |
| DeepSeek V4 Flash 0731 | DeepSeek | 21 | Sail Research (fp4) 0.04% | Alibaba Cloud Int. 15.01% | `sail-research/fp4, sail-research/us, wafer/fast, relace/fp4`; ignore `deepinfra/fp8, mancer/fp8, phala, reka, alibaba` | [JSON](models/deepseek-v4-flash-0731.json) |
| GLM 5.3 | Z.ai | 37 | Amazon Bedrock 0.08% | GMICloud (fp8) 23.02% | `amazon-bedrock/us-east-1, friendli, inference-net/fp4, inference-net/us`; ignore `digitalocean, baidu/fp8, phala, atlas-cloud/fp8, novita/fp8, gmicloud/fp8` | [JSON](models/glm-5.3.json) |
| MiMo-V2.6-Flash | Xiaomi | 9 | Venice (fp8) 1.41% | io.net (fp8) 4.73% | `venice/fp8, xiaomi/fp8, deepinfra/fp8`; ignore `makora/fp8, relace/mxfp4, io-net/fp8` | [JSON](models/mimo-v2.6-flash.json) |
| Hy3 | Tencent | 5 | Tencent Cloud (fp8) 1.46% | NovitaAI 3.23% | `tencent/fp8, atlas-cloud/fp8, gmicloud/bf16`; ignore `novita` | [JSON](models/hy3.json) |
| DeepSeek V4 Flash 0423 | DeepSeek | 16 | Crusoe (fp8) 0.46% | Azure 5.81% | `crusoe/fp8, wafer/mxfp4, relace/fp4`; ignore `baidu/fp8, atlas-cloud/fp4, venice, azure/us` | [JSON](models/deepseek-v4-flash.json) |
| MiniMax M3.1 Flash | MiniMax | 11 | Together 1.17% | Fireworks 11.02% | `together, parasail/fp8, modelrun/fp4`; ignore `gmicloud/fp8, fireworks` | [JSON](models/minimax-m3.1-flash.json) |
| Kimi K3 | MoonshotAI | 20 | DeepInfra (mxfp4) 0.01% | Modal (mxfp4) 0.63% | `deepinfra/mxfp4, fireworks/spot, fireworks`; ignore `—` | [JSON](models/kimi-k3.json) |
| GLM 5.2 | Z.ai | 23 | Friendli 0.15% | Mistral 5.14% | `friendli, wafer, morph/fp8`; ignore `atlas-cloud/fp8, mistral/eu` | [JSON](models/glm-5.2.json) |
| MiMo-V2.6-Pro | Xiaomi | 4 | DeepInfra (fp8) 1.48% | NovitaAI (fp8) 5.47% | `deepinfra/fp8`; ignore `gmicloud/bf16, xiaomi/fp8, streamlake, novita/fp8` | [JSON](models/mimo-v2.6-pro.json) |
| MiniMax M3 | MiniMax | 11 | ModelRun [by Modular] (fp8) 0.27% | AtlasCloud (fp8) 36.08% | `modelrun/fp8, venice/fp8, novita/fp8`; ignore `mara, coreweave/fp4, gmicloud/fp8, atlas-cloud/fp8` | [JSON](models/minimax-m3.json) |
| Qwen3.8 27B | Qwen | 21 | Darkbloom (fp4) 0.07% | Cerebras (fp16) 13.98% | `darkbloom/fp4, venice/fp8, modelrun/fp4`; ignore `novita, cerebras/fp16` | [JSON](models/qwen3.8-27b.json) |
| DeepSeek V4 Pro 0813 | DeepSeek | 21 | Ionstream 0.04% | Cloudflare 7.93% | `ionstream, parasail/fp8, atlas-cloud/fp8`; ignore `digitalocean, gmicloud/fp8, cloudflare` | [JSON](models/deepseek-v4-pro-0813.json) |
| DeepSeek V4 Pro 0423 | DeepSeek | 13 | Cloudflare 0.78% | Baidu Qianfan (fp8) 6.20% | `cloudflare, siliconflow/fp8, crusoe/fp8`; ignore `alibaba/fp8, atlas-cloud/fp4, digitalocean, azure/us, baidu/fp8` | [JSON](models/deepseek-v4-pro.json) |
| gpt-oss-120b | OpenAI | 15 | Groq 0.04% | Nebius Token Factory (fp4) 80.12% | `groq, deepinfra/bf16, coreweave/fp4`; ignore `akashml/bf16, parasail/fp4, dekallm/bf16, mancer/fp8, crusoe/bf16, phala, baseten/fp4, near-ai/fp4, venice/int4, novita/fp4, cerebras/fp16, sail-research/fp4, nebius/fp4` | [JSON](models/gpt-oss-120b.json) |
| gpt-oss-20b | OpenAI | 7 | Darkbloom (fp8) 0.13% | Parasail (fp4) 22.66% | `darkbloom/fp8, groq, deepinfra/bf16`; ignore `coreweave/fp4, akashml/fp4, parasail/fp4` | [JSON](models/gpt-oss-20b.json) |
| Qwen3.5-9B | Qwen | 5 | Darkbloom (fp4) 0.08% | SiliconFlow (fp8) 89.86% | `darkbloom/fp4, together, primeintellect/bf16`; ignore `siliconflow/fp8` | [JSON](models/qwen3.5-9b.json) |
| Gemma 4 31B | Google | 11 | ModelRun [by Modular] (fp4) 0.09% | SiliconFlow (fp8) 5.88% | `modelrun/fp4, friendli, novita/bf16`; ignore `sambanova, siliconflow/fp8, dekallm` | [JSON](models/gemma-4-31b-it.json) |
| Gemma 4 26B A4B | Google | 11 | NextBit (bf16) 0.00% | AkashML (fp4) 8.70% | `nextbit/bf16, darkbloom, primeintellect/bf16`; ignore `dekallm/bf16, google-vertex/global, cloudflare, akashml/fp4` | [JSON](models/gemma-4-26b-a4b-it.json) |
| Nemotron 3.5 Super VL | NVIDIA | 8 | Together 0.66% | CoreWeave (nvfp4) 8.36% | `together, io-net/nvfp4, nebius/bf16`; ignore `venice/fp4, crusoe/bf16, baseten/fp4, deepinfra/fp4, coreweave/nvfp4` | [JSON](models/nemotron-3.5-super-vl.json) |
| Kimi K2.6 | MoonshotAI | 14 | SiliconFlow (fp8) 0.06% | Moonshot AI 2.92% | `siliconflow/fp8, novita, inceptron/int4`; ignore `—` | [JSON](models/kimi-k2.6.json) |
| Qwen3.6 35B A3B | Qwen | 6 | AkashML (fp8) 1.63% | Phala 5.75% | `akashml/fp8, parasail/fp8, deepinfra/fp8`; ignore `dekallm, phala` | [JSON](models/qwen3.6-35b-a3b.json) |
| Beam 1.0 501B A23B | Reflection | 3 | Together 1.01% | CoreWeave (fp4) 6.52% | `together, nebius/fp4`; ignore `coreweave/fp4` | [JSON](models/beam-1.0-501b-a23b.json) |
| Qwen3 235B A22B Instruct 2507 | Qwen | 3 | Parasail (fp8) 0.78% | Venice (fp8) 2.21% | `parasail/fp8, deepinfra/fp8, venice/fp8`; ignore `—` | [JSON](models/qwen3-235b-a22b-2507.json) |
| Nemotron 3.5 Lightning | NVIDIA | 4 | Darkbloom (int4) 0.01% | DeepInfra (bf16) 9.96% | `darkbloom/int4`; ignore `io-net, coreweave/bf16, deepinfra/bf16` | [JSON](models/nemotron-3.5-lightning.json) |
| DeepSeek V3.2 | DeepSeek | 4 | SiliconFlow (fp8) 0.60% | GMICloud (fp8) 1.61% | `siliconflow/fp8, deepinfra/fp4, phala`; ignore `—` | [JSON](models/deepseek-v3.2.json) |
| Muse Spark Open | Meta | 6 | Nebius Token Factory (fp4) 0.07% | Modal 3.26% | `nebius/fp4, fireworks, together`; ignore `modal` | [JSON](models/muse-spark-open.json) |
| DeepSeek V4 Flash Vision Exp | DeepSeek | 3 | GMICloud (fp8) 0.58% | DeepInfra (fp8) 1.92% | `gmicloud/fp8, siliconflow/fp8, deepinfra/fp8`; ignore `—` | [JSON](models/deepseek-v4-flash-vision-exp.json) |
| Llama 3.3 70B Instruct | Meta | 5 | Groq 2.21% | AkashML (fp8) 32.75% | `groq`; ignore `coreweave/fp16, deepinfra/turbo, novita/bf16, akashml/fp8` | [JSON](models/llama-3.3-70b-instruct.json) |
| Kimi K2.7 Code | MoonshotAI | 10 | NovitaAI (int4) 0.03% | CoreWeave (int4) 3.35% | `novita/int4, streamlake, inceptron/int4`; ignore `coreweave/int4` | [JSON](models/kimi-k2.7-code.json) |
| Qwen3 30B A3B Instruct 2507 | Qwen | 2 | Nebius Token Factory (fp8) 2.89% | SiliconFlow (fp8) 4.38% | `nebius/fp8`; ignore `siliconflow/fp8, dekallm` | [JSON](models/qwen3-30b-a3b-instruct-2507.json) |
| Nemotron 3 Ultra | NVIDIA | 3 | DeepInfra (fp4/flex) 0.17% | Baseten (fp4) 3.58% | `deepinfra/fp4/flex, venice/fp8, deepinfra/fp4`; ignore `baseten/fp4` | [JSON](models/nemotron-3-ultra-550b-a55b.json) |
| MiniMax M2.7 | MiniMax | 5 | Groq 0.32% | NovitaAI (fp8) 3.94% | `groq, minimax, atlas-cloud/fp8`; ignore `gmicloud/fp8, novita/fp8` | [JSON](models/minimax-m2.7.json) |
| Qwen3.5 397B A17B | Qwen | 7 | Alibaba Cloud Int. 0.58% | Phala 11.30% | `alibaba, novita, deepinfra/fp8`; ignore `phala` | [JSON](models/qwen3.5-397b-a17b.json) |
| Qwen3.8 2.4T A95B | Qwen | 5 | SiliconFlow (fp8) 0.19% | Together 4.19% | `siliconflow/fp8, venice, alibaba`; ignore `novita, together` | [JSON](models/qwen3.8-2.4t-a95b.json) |
| Kimi K2.5 | MoonshotAI | 5 | SiliconFlow (int4) 0.17% | Amazon Bedrock 6.65% | `siliconflow/int4, atlas-cloud/int4, venice`; ignore `novita, amazon-bedrock/us-east-2` | [JSON](models/kimi-k2.5.json) |
| GLM 5.1 | Z.ai | 7 | Friendli 0.00% | StreamLake (fp8) 6.96% | `friendli, alibaba/fp8, chutes/fp8`; ignore `phala, baidu/fp8, streamlake/fp8` | [JSON](models/glm-5.1.json) |
| MiMo-V2.5-Pro | Xiaomi | 2 | AtlasCloud (fp8) 2.04% | DigitalOcean 2.98% | `atlas-cloud/fp8, digitalocean`; ignore `—` | [JSON](models/mimo-v2.5-pro.json) |
| GLM 5 | Z.ai | 5 | GMICloud (fp8) 0.74% | Amazon Bedrock 3.13% | `gmicloud/fp8, streamlake/fp8, z-ai/fp8`; ignore `amazon-bedrock` | [JSON](models/glm-5.json) |
| Llama 3.1 8B Instruct | Meta | 2 | Groq 1.00% | CoreWeave (bf16) 67.14% | `groq`; ignore `coreweave/bf16` | [JSON](models/llama-3.1-8b-instruct.json) |
| DeepSeek V3.1 | DeepSeek | 3 | SiliconFlow (fp8) 1.03% | DeepInfra (fp4) 4.03% | `siliconflow/fp8, coreweave/fp8`; ignore `deepinfra/fp4` | [JSON](models/deepseek-chat-v3.1.json) |
| Granite 4.2 8B | IBM | 2 | DeepInfra (bf16) 2.00% | CoreWeave (bf16) 3.71% | `deepinfra/bf16`; ignore `coreweave/bf16` | [JSON](models/granite-4.2-8b.json) |
| Mistral Nemo | Mistral | 2 | Mistral 0.62% | io.net (fp16) 1.38% | `mistral/eu, io-net/fp16`; ignore `deepinfra/fp8` | [JSON](models/mistral-nemo.json) |
| GLM 4.7 | Z.ai | 3 | Z.ai (fp4) 0.93% | DeepInfra (fp4) 2.22% | `z-ai/fp4, novita/fp8, deepinfra/fp4`; ignore `—` | [JSON](models/glm-4.7.json) |
| Gemma 3 27B | Google | 0 | — | — | none (single provider) | [JSON](models/gemma-3-27b-it.json) |

## Model makers

The worst error rate on each maker's models.

- DeepSeek: 15.01% of DeepSeek V4 Flash 0731's tool calls fail on Alibaba Cloud Int.
- Google: 8.70% of Gemma 4 26B A4B's tool calls fail on AkashML (fp4).
- IBM: 3.71% of Granite 4.2 8B's tool calls fail on CoreWeave (bf16).
- Meta: 67.14% of Llama 3.1 8B Instruct's tool calls fail on CoreWeave (bf16).
- MiniMax: 36.08% of MiniMax M3's tool calls fail on AtlasCloud (fp8).
- Mistral: 1.38% of Mistral Nemo's tool calls fail on io.net (fp16).
- MoonshotAI: 6.65% of Kimi K2.5's tool calls fail on Amazon Bedrock.
- NVIDIA: 9.96% of Nemotron 3.5 Lightning's tool calls fail on DeepInfra (bf16).
- OpenAI: 80.12% of gpt-oss-120b's tool calls fail on Nebius Token Factory (fp4).
- Qwen: 89.86% of Qwen3.5-9B's tool calls fail on SiliconFlow (fp8).
- Reflection: 6.52% of Beam 1.0 501B A23B's tool calls fail on CoreWeave (fp4).
- Tencent: 3.23% of Hy3's tool calls fail on NovitaAI.
- Xiaomi: 5.47% of MiMo-V2.6-Pro's tool calls fail on NovitaAI (fp8).
- Z.ai: 23.02% of GLM 5.3's tool calls fail on GMICloud (fp8).

## Providers

The same provider can handle tool calls well on one model and struggle on the next: each provider's error rate from its best model to its worst, each variant on its own, over the last 7 days, on models two providers or more serve, for every provider on two of those models or more, A to Z; each provider's rate on each model is in data.json.

- [AkashML](p/akashml/): 0.52% to 32.75% across 8 models
- [Alibaba Cloud Int.](p/alibaba/): 0.05% to 15.01% across 14 models
- [Amazon Bedrock](p/amazon-bedrock/): 0.05% to 6.65% across 5 models
- [AtlasCloud](p/atlas-cloud/): 0.30% to 36.08% across 16 models
- [Azure](p/azure/): 4.07% to 5.81% across 2 models
- [Baidu Qianfan](p/baidu/): 0.50% to 6.20% across 10 models
- [Baseten](p/baseten/): 0.06% to 9.87% across 11 models
- [Cerebras](p/cerebras/): 13.98% to 36.42% across 2 models
- [Chutes](p/chutes/): 0.87% to 2.07% across 4 models
- [Cloudflare](p/cloudflare/): 0.40% to 7.93% across 10 models
- [Cohere](p/cohere/): 0.17% to 2.70% across 6 models
- [CoreWeave](p/coreweave/): 0.29% to 67.14% across 20 models
- [Crusoe](p/crusoe/): 0.46% to 7.33% across 10 models
- [Darkbloom](p/darkbloom/): 0.00% to 0.13% across 5 models
- [Decart](p/decart/): 0.34% to 1.47% across 5 models
- [DeepInfra](p/deepinfra/): 0.01% to 9.96% across 33 models
- [DeepSeek](p/deepseek/): 1.05% to 2.01% across 2 models
- [DekaLLM](p/dekallm/): 0.40% to 3.94% across 6 models
- [DigitalOcean](p/digitalocean/): 0.20% to 3.81% across 13 models
- [Fireworks](p/fireworks/): 0.01% to 11.02% across 8 models
- [Friendli](p/friendli/): 0.00% to 0.80% across 5 models
- [GMICloud](p/gmicloud/): 0.58% to 23.02% across 17 models
- [Google Vertex](p/google-vertex/): 0.50% to 5.30% across 2 models
- [Groq](p/groq/): 0.04% to 2.21% across 5 models
- [Inceptron](p/inceptron/): 0.22% to 0.56% across 6 models
- [inference.net](p/inference-net/): 0.06% to 1.74% across 6 models
- [io.net](p/io-net/): 0.86% to 4.73% across 6 models
- [Ionstream](p/ionstream/): 0.04% to 1.49% across 3 models
- [Makora](p/makora/): 0.32% to 3.92% across 8 models
- [MiniMax](p/minimax/): 1.76% to 2.98% across 3 models
- [Mistral](p/mistral/): 0.62% to 5.14% across 4 models
- [Modal](p/modal/): 0.20% to 3.26% across 7 models
- [ModelRun [by Modular]](p/modelrun/): 0.09% to 1.42% across 5 models
- [Moonshot AI](p/moonshotai/): 0.22% to 2.92% across 3 models
- [Morph](p/morph/): 0.07% to 0.44% across 6 models
- [NEAR AI](p/near-ai/): 0.94% to 10.37% across 3 models
- [Nebius Token Factory](p/nebius/): 0.07% to 80.12% across 12 models
- [NextBit](p/nextbit/): 0.00% to 0.00% across 2 models
- [NovitaAI](p/novita/): 0.03% to 29.28% across 26 models
- [Open Inference](p/open-inference/): 0.32% to 1.46% across 3 models
- [Parasail](p/parasail/): 0.21% to 22.66% across 19 models
- [Particle AI](p/particle-ai/): 0.23% to 0.34% across 2 models
- [Phala](p/phala/): 0.17% to 11.30% across 12 models
- [Prime Intellect](p/primeintellect/): 0.02% to 0.58% across 6 models
- [Reka AI](p/reka/): 0.18% to 10.07% across 6 models
- [Relace](p/relace/): 0.05% to 4.17% across 11 models
- [Sail Research](p/sail-research/): 0.03% to 1.06% across 7 models
- [SambaNova](p/sambanova/): 2.53% to 5.05% across 3 models
- [SiliconFlow](p/siliconflow/): 0.06% to 89.86% across 22 models
- [StreamLake](p/streamlake/): 0.09% to 6.96% across 14 models
- [Tencent Cloud](p/tencent/): 0.16% to 1.46% across 2 models
- [Tenstorrent](p/tenstorrent/): 1.19% to 2.77% across 2 models
- [Together](p/together/): 0.25% to 4.19% across 15 models
- [Venice](p/venice/): 0.19% to 17.23% across 19 models
- [Wafer](p/wafer/): 0.12% to 1.06% across 10 models
- [Xiaomi](p/xiaomi/): 1.71% to 4.59% across 2 models
- [Z.ai](p/z-ai/): 0.65% to 2.22% across 6 models

## Structured outputs

> On a typical open-weight model, the worst provider on OpenRouter breaks structured outputs 12× as often as the best.

The same providers, measured on OpenRouter's published structured-output error rate per provider endpoint: of the requests that asked for a structured output (`response_format`), the share whose answer OpenRouter counts as failed (it doesn't document what fails one), over the last 7 days, with the same settings decided the same way: 39 models, 55 providers; each model's `structured_outputs` in its JSON. Implementations diverge on strings, optionals, composition and conditionals (https://dottxt.ai/announcements/dottxt-api): the schema one provider enforces, another ignores.

| Model | Maker | Providers | Most reliable | Least reliable | Routing | Data |
|---|---|---|---|---|---|---|
| GLM 5.3 Flash | Z.ai | 29 | Parasail (fp4) 0.34% | Mistral 63.40% | `parasail/fp4, fireworks, baseten/fp8`; ignore `phala/nvfp4, darkbloom/fp4, cohere, mistral` | [JSON](models/glm-5.3-flash.json) |
| DeepSeek V4.1 Flash | DeepSeek | 25 | DeepInfra (fp8/flex) 0.26% | Cohere 4.28% | `deepinfra/fp8/flex, baseten/fast, deepinfra/fp8/priority`; ignore `coreweave/fp8, google-vertex/global, mancer/fp8, crusoe/fp8, cohere` | [JSON](models/deepseek-v4.1-flash.json) |
| DeepSeek V4 Flash 0731 | DeepSeek | 17 | AtlasCloud (fp4) 0.25% | Baidu Qianfan (fp8) 7.65% | `atlas-cloud/fp4, mancer/fp8, deepinfra/fp8`; ignore `cohere, phala, baidu/fp8` | [JSON](models/deepseek-v4-flash-0731.json) |
| DeepSeek V4 Flash 0423 | DeepSeek | 13 | Alibaba Cloud Int. (fp8) 0.60% | AtlasCloud (fp4) 36.71% | `alibaba/fp8/us-east-1, deepinfra/fp8, streamlake/fp8`; ignore `venice, cloudflare, digitalocean, atlas-cloud/fp4` | [JSON](models/deepseek-v4-flash.json) |
| Gemma 4 26B A4B | Google | 11 | Venice (bf16) 0.38% | Google Vertex 7.85% | `venice/bf16, coreweave/bf16, deepinfra/fp8`; ignore `darkbloom, google-vertex/global` | [JSON](models/gemma-4-26b-a4b-it.json) |
| DeepSeek V4.1 Flash 1017 | DeepSeek | 28 | Reka AI 0.34% | Phala 11.35% | `reka, parasail/fp8, inference-net/fp8, inference-net/us`; ignore `siliconflow/fp8, open-inference/fp4, phala` | [JSON](models/deepseek-v4.1-flash-1017.json) |
| Gemma 4 31B | Google | 11 | ModelRun [by Modular] (fp4) 0.37% | Venice (fp4) 3.51% | `modelrun/fp4, friendli, parasail/fp8`; ignore `venice/fp4` | [JSON](models/gemma-4-31b-it.json) |
| gpt-oss-120b | OpenAI | 18 | MARA 0.68% | Google Vertex 14.30% | `mara, akashml/bf16, nebius/fp4`; ignore `siliconflow/fp8, coreweave/fp4, sail-research/fp4, google-vertex/global` | [JSON](models/gpt-oss-120b.json) |
| Qwen3.6 35B A3B | Qwen | 8 | Parasail (fp8) 0.15% | Darkbloom (fp4) 4.17% | `parasail/fp8, akashml/fp8, siliconflow/fp8`; ignore `darkbloom/fp4` | [JSON](models/qwen3.6-35b-a3b.json) |
| gpt-oss-20b | OpenAI | 8 | CoreWeave (fp4) 0.92% | Darkbloom (fp8) 23.65% | `coreweave/fp4, groq`; ignore `deepinfra/bf16, akashml/fp4, parasail/fp4, siliconflow/fp8, dekallm/bf16, darkbloom/fp8` | [JSON](models/gpt-oss-20b.json) |
| GLM 5.3 | Z.ai | 25 | Together 0.24% | Friendli 10.26% | `together, amazon-bedrock/us-east-1, baseten/fp4`; ignore `sail-research/fp8, sail-research/us, cloudflare, digitalocean, crusoe/fp4, nebius/fp4, parasail/fp8, modelrun/fp4, coreweave/fp8, inference-net/fp4, inference-net/us, baidu/fp8, reka, primeintellect, dekallm, friendli, venice, atlas-cloud/fp8` | [JSON](models/glm-5.3.json) |
| Qwen3.8 27B | Qwen | 16 | NEAR AI (fp8) 0.37% | Reka AI (fp8) 4.50% | `near-ai/fp8, akashml/fp8, modelrun/fp4`; ignore `wafer/fp8, cloudflare, modal, reka/fp8` | [JSON](models/qwen3.8-27b.json) |
| Mistral Nemo | Mistral | 3 | Parasail (fp8) 0.70% | DekaLLM (fp8) 3.37% | `parasail/fp8, deepinfra/fp8`; ignore `dekallm/fp8, mistral/eu` | [JSON](models/mistral-nemo.json) |
| Gemma 3 27B | Google | 4 | NextBit (fp8) 0.96% | Nebius Token Factory (fp8) 1.98% | `nextbit/fp8, deepinfra/fp8, parasail/fp8`; ignore `—` | [JSON](models/gemma-3-27b-it.json) |
| GLM 5.2 | Z.ai | 19 | Venice (fp8) 0.20% | DigitalOcean 47.47% | `venice/fp8, mistral/eu, together`; ignore `baseten/fp8, deepinfra/fp4/flex, deepinfra/fp4, nebius/fp4, mistral/zdr, wafer, alibaba/fast, gmicloud/fp8, digitalocean` | [JSON](models/glm-5.2.json) |
| MiMo-V2.6-Flash | Xiaomi | 8 | GMICloud (bf16) 0.19% | io.net (fp8) 8.15% | `gmicloud/bf16, makora/fp8, deepinfra/fp8`; ignore `darkbloom/fp4, io-net/fp8` | [JSON](models/mimo-v2.6-flash.json) |
| Llama 3.3 70B Instruct | Meta | 5 | SambaNova Turbo 0.14% | DeepInfra (fp8/turbo) 4.22% | `sambanova-turbo, parasail/fp8, akashml/fp8`; ignore `deepinfra/turbo` | [JSON](models/llama-3.3-70b-instruct.json) |
| Qwen3 235B A22B Instruct 2507 | Qwen | 3 | DeepInfra (fp8) 1.33% | Venice (fp8) 1.35% | `deepinfra/fp8, parasail/fp8, venice/fp8`; ignore `—` | [JSON](models/qwen3-235b-a22b-2507.json) |
| DeepSeek V3.2 | DeepSeek | 5 | DeepInfra (fp4) 0.87% | Baidu Qianfan (fp8) 7.12% | `deepinfra/fp4, atlas-cloud/fp8, phala`; ignore `baidu/fp8` | [JSON](models/deepseek-v3.2.json) |
| Qwen3 30B A3B Instruct 2507 | Qwen | 3 | Nebius Token Factory (fp8) 0.99% | DekaLLM 1.49% | `nebius/fp8, siliconflow/fp8, dekallm`; ignore `—` | [JSON](models/qwen3-30b-a3b-instruct-2507.json) |
| Qwen3.5-9B | Qwen | 6 | Prime Intellect (bf16) 0.34% | Together 79.14% | `primeintellect/bf16, deepinfra/bf16`; ignore `parasail/bf16, siliconflow/fp8, darkbloom/fp4, together` | [JSON](models/qwen3.5-9b.json) |
| Kimi K3 | MoonshotAI | 15 | DeepInfra (mxfp4) 0.19% | Modal (mxfp4) 13.30% | `deepinfra/mxfp4, parasail/fp4, makora/fp4`; ignore `phala, morph/fp8, wafer/fast, akashml/fp4, pareto-alpha/us, sail-research/fp4, inference-net/fp4, inference-net/us, modal/mxfp4` | [JSON](models/kimi-k3.json) |
| DeepSeek V4 Pro 0423 | DeepSeek | 8 | DeepInfra (fp8) 0.34% | Cloudflare 14.75% | `deepinfra/fp8, venice, baidu/fp8`; ignore `streamlake/fp8, atlas-cloud/fp4, crusoe/fp8, alibaba/fp8, cloudflare` | [JSON](models/deepseek-v4-pro.json) |
| DeepSeek V4 Pro 0813 | DeepSeek | 12 | Together 0.53% | Alibaba Cloud Int. 5.59% | `together, ionstream, cloudflare`; ignore `digitalocean, atlas-cloud/fp8, venice, coreweave/fp8, alibaba` | [JSON](models/deepseek-v4-pro-0813.json) |
| Kimi K2.6 | MoonshotAI | 13 | Cloudflare 1.10% | Chutes (int4) 3.28% | `cloudflare, parasail/int4, venice/int4`; ignore `chutes/int4, phala` | [JSON](models/kimi-k2.6.json) |
| DeepSeek V3.1 | DeepSeek | 4 | MARA 0.16% | SiliconFlow (fp8) 4.84% | `mara, coreweave/fp8, deepinfra/fp4`; ignore `siliconflow/fp8` | [JSON](models/deepseek-chat-v3.1.json) |
| MiMo-V2.6-Pro | Xiaomi | 3 | DeepInfra (fp8) 2.79% | GMICloud (bf16) 3.23% | `deepinfra/fp8`; ignore `xiaomi/fp8, gmicloud/bf16` | [JSON](models/mimo-v2.6-pro.json) |
| Nemotron 3.5 Lightning | NVIDIA | 4 | CoreWeave (bf16) 1.63% | Darkbloom (int4) 13.31% | `coreweave/bf16, deepinfra/bf16`; ignore `io-net, darkbloom/int4` | [JSON](models/nemotron-3.5-lightning.json) |
| Nemotron 3.5 Super VL | NVIDIA | 7 | Nebius Token Factory (bf16) 1.45% | Darkbloom (nvfp4) 9.30% | `nebius/bf16, venice/fp4, coreweave/nvfp4`; ignore `deepinfra/fp4, together, darkbloom/nvfp4` | [JSON](models/nemotron-3.5-super-vl.json) |
| MiniMax M3 | MiniMax | 3 | CoreWeave (fp4) 0.49% | ModelRun [by Modular] (fp8) 3.76% | `coreweave/fp4, together`; ignore `modelrun/fp8` | [JSON](models/minimax-m3.json) |
| Qwen3.5 397B A17B | Qwen | 7 | Phala 0.54% | StreamLake 2.33% | `phala, venice, deepinfra/fp8`; ignore `—` | [JSON](models/qwen3.5-397b-a17b.json) |
| MiniMax M3.1 Flash | MiniMax | 5 | ModelRun [by Modular] (fp4) 0.13% | Parasail (fp8) 1.05% | `modelrun/fp4, inference-net/fp4, digitalocean`; ignore `fireworks, sambanova` | [JSON](models/minimax-m3.1-flash.json) |
| Kimi K2.5 | MoonshotAI | 3 | SiliconFlow (int4) 2.22% | Venice 10.10% | `siliconflow/int4, atlas-cloud/int4`; ignore `venice, novita` | [JSON](models/kimi-k2.5.json) |
| Hy3 | Tencent | 3 | AtlasCloud (fp8) 1.05% | NovitaAI 5.29% | `atlas-cloud/fp8, tencent/fp8`; ignore `novita, phala` | [JSON](models/hy3.json) |
| Beam 1.0 501B A23B | Reflection | 2 | Nebius Token Factory (fp4) 1.65% | CoreWeave (fp4) 2.16% | `nebius/fp4, coreweave/fp4`; ignore `digitalocean, baseten/fp4` | [JSON](models/beam-1.0-501b-a23b.json) |
| Qwen3.8 2.4T A95B | Qwen | 3 | Together 0.38% | NovitaAI 70.75% | `together`; ignore `siliconflow/fp8, alibaba, novita` | [JSON](models/qwen3.8-2.4t-a95b.json) |
| Muse Spark Open | Meta | 4 | Together 0.77% | Makora (fp4) 3.56% | `together, nebius/fp4, modal`; ignore `makora/fp4, fireworks` | [JSON](models/muse-spark-open.json) |
| GLM 5.1 | Z.ai | 3 | Nebius Token Factory (fp8) 0.46% | Baidu Qianfan (fp8) 3.30% | `nebius/fp8, streamlake/fp8`; ignore `baidu/fp8, alibaba/fp8, chutes/fp8, atlas-cloud/fp8` | [JSON](models/glm-5.1.json) |
| GLM 5 | Z.ai | 3 | Baidu Qianfan (fp8) 3.25% | GMICloud (fp8) 66.31% | none (no reliable provider) | [JSON](models/glm-5.json) |

The worst error rate on each maker's models.

- DeepSeek: 36.71% of DeepSeek V4 Flash 0423's structured outputs fail on AtlasCloud (fp4).
- Google: 7.85% of Gemma 4 26B A4B's structured outputs fail on Google Vertex.
- Meta: 4.22% of Llama 3.3 70B Instruct's structured outputs fail on DeepInfra (fp8/turbo).
- MiniMax: 3.76% of MiniMax M3's structured outputs fail on ModelRun [by Modular] (fp8).
- Mistral: 3.37% of Mistral Nemo's structured outputs fail on DekaLLM (fp8).
- MoonshotAI: 13.30% of Kimi K3's structured outputs fail on Modal (mxfp4).
- NVIDIA: 13.31% of Nemotron 3.5 Lightning's structured outputs fail on Darkbloom (int4).
- OpenAI: 23.65% of gpt-oss-20b's structured outputs fail on Darkbloom (fp8).
- Qwen: 79.14% of Qwen3.5-9B's structured outputs fail on Together.
- Reflection: 2.16% of Beam 1.0 501B A23B's structured outputs fail on CoreWeave (fp4).
- Tencent: 5.29% of Hy3's structured outputs fail on NovitaAI.
- Xiaomi: 8.15% of MiMo-V2.6-Flash's structured outputs fail on io.net (fp8).
- Z.ai: 66.31% of GLM 5's structured outputs fail on GMICloud (fp8).

Each provider's structured-output error rate from its best model to its worst, as above:

- [AkashML](json/p/akashml/): 0.16% to 6.59% across 8 models
- [Alibaba Cloud Int.](json/p/alibaba/): 0.60% to 22.75% across 6 models
- [AtlasCloud](json/p/atlas-cloud/): 0.25% to 36.71% across 11 models
- [Baidu Qianfan](json/p/baidu/): 0.61% to 7.65% across 12 models
- [Baseten](json/p/baseten/): 0.28% to 3.88% across 5 models
- [Chutes](json/p/chutes/): 1.12% to 3.28% across 3 models
- [Cloudflare](json/p/cloudflare/): 1.04% to 14.75% across 7 models
- [Cohere](json/p/cohere/): 2.24% to 7.04% across 5 models
- [CoreWeave](json/p/coreweave/): 0.25% to 7.77% across 19 models
- [Crusoe](json/p/crusoe/): 1.26% to 4.87% across 10 models
- [Darkbloom](json/p/darkbloom/): 2.29% to 23.65% across 9 models
- [Decart](json/p/decart/): 1.09% to 2.96% across 5 models
- [DeepInfra](json/p/deepinfra/): 0.19% to 4.22% across 29 models
- [DekaLLM](json/p/dekallm/): 0.93% to 12.86% across 9 models
- [DigitalOcean](json/p/digitalocean/): 0.80% to 47.47% across 9 models
- [Fireworks](json/p/fireworks/): 0.37% to 2.19% across 6 models
- [Friendli](json/p/friendli/): 0.39% to 10.26% across 4 models
- [GMICloud](json/p/gmicloud/): 0.19% to 66.31% across 4 models
- [Google Vertex](json/p/google-vertex/): 3.71% to 14.30% across 3 models
- [Groq](json/p/groq/): 2.02% to 2.70% across 2 models
- [Inceptron](json/p/inceptron/): 1.43% to 2.66% across 4 models
- [inference.net](json/p/inference-net/): 0.54% to 5.74% across 6 models
- [io.net](json/p/io-net/): 5.17% to 8.15% across 2 models
- [Ionstream](json/p/ionstream/): 0.56% to 1.86% across 3 models
- [Makora](json/p/makora/): 0.48% to 3.56% across 8 models
- [Mancer](json/p/mancer/): 0.29% to 2.11% across 4 models
- [MARA](json/p/mara/): 0.16% to 0.68% across 2 models
- [Mistral](json/p/mistral/): 0.28% to 63.40% across 3 models
- [Modal](json/p/modal/): 1.12% to 13.30% across 7 models
- [ModelRun [by Modular]](json/p/modelrun/): 0.13% to 4.42% across 5 models
- [Moonshot AI](json/p/moonshotai/): 1.62% to 2.33% across 2 models
- [Morph](json/p/morph/): 1.16% to 3.15% across 6 models
- [NEAR AI](json/p/near-ai/): 0.37% to 2.16% across 3 models
- [Nebius Token Factory](json/p/nebius/): 0.46% to 4.92% across 12 models
- [NextBit](json/p/nextbit/): 0.45% to 0.96% across 3 models
- [NovitaAI](json/p/novita/): 0.90% to 70.75% across 4 models
- [Open Inference](json/p/open-inference/): 1.05% to 3.57% across 3 models
- [Parasail](json/p/parasail/): 0.15% to 7.33% across 23 models
- [Particle AI](json/p/particle-ai/): 1.38% to 1.75% across 2 models
- [Phala](json/p/phala/): 0.54% to 11.35% across 9 models
- [Prime Intellect](json/p/primeintellect/): 0.34% to 7.77% across 6 models
- [Reka AI](json/p/reka/): 0.34% to 6.85% across 6 models
- [Relace](json/p/relace/): 1.76% to 1.79% across 2 models
- [Sail Research](json/p/sail-research/): 0.71% to 11.78% across 8 models
- [SiliconFlow](json/p/siliconflow/): 0.82% to 11.45% across 13 models
- [StreamLake](json/p/streamlake/): 1.22% to 6.63% across 9 models
- [Together](json/p/together/): 0.24% to 79.14% across 14 models
- [Venice](json/p/venice/): 0.20% to 10.09% across 15 models
- [Wafer](json/p/wafer/): 0.77% to 5.94% across 10 models
- [Xiaomi](json/p/xiaomi/): 2.82% to 3.15% across 2 models

dottxt makes invalid tool calls and broken structured outputs impossible: https://dottxt.ai