The author of Olly, an AI assistant that has transacted over 6 million messages via OpenRouter, describes on his blog several of the pitfalls that plague users of AI inference hub OpenRouter:
“The same model will benchmark very differently”: OpenRouter itself publishes results for GPQA-Diamond and TAU-Bench (under “AutoExacto Benchmarks”) - and they vary wildly among the providers:
Chart courtesy mmoustafa.com Vision performance is uneven as well: “DeepInfra's Qwen endpoint read a K as an R, called red blue, and described the word "umbrella" as "funny", while four other hosts of the same weights got everything right. Venice and Together didn't see the MiniMax images at all. The model page says it supports image input, but two of its providers don't and even worse they'll pretend everything is 200 OK.”
Reasoning effort doesn’t mean the same thing: according to this chart, there are no two providers where a particular effort setting would result in the same effort being spent:
Chart courtesy mmoustafa.com Quantization filters don’t buy you quality: precision is a bad proxy for quality - “The fp4 hosts land in the middle of the fp8 pack. The three worst GPQA scores on DeepSeek are one of each: an fp4 host, an fp8 host, and one that declares nothing. GLM's best scorer on both benchmarks, Wafer, declares nothing at all.”
… and several technical inconsistencies and errors more
Reactions
Commenters on Hacker News largely confirmed the findings.
Several pointed out that quantization alone cannot explain these differences. OpenRouter’s COO writes in that inference providers have “a huge amount of code” between weights and API, and that OpenRouter continuously benchmarks endpoints to detect degradation.
Another practical problem - not mentioned in the original blog post - is caching. OpenRouter publishes a cache hit rate for individual providers. For Kimi K3, for example, its current provider table shows substantially different hit rates and consequently different effective input prices despite similar listed prices. So automatically choosing the nominally cheapest provider need not result in the cheapest request. Further, prompt caches usually live at the provider, so switching providers can turn the next request into a cache miss. This matters particularly for agents and long conversations that repeatedly send large system prompts, tool definitions and history. OpenRouter uses sticky routing to improve cache hits, but a failover can still lose the warm cache.
My own experience
Reproducibility
One commenter responded “We're using OpenRouter for some research tasks and it's making me question everything” and another noted that the open weights ecosystem had unremediated reproducibility issues. I too ran into the reproducibility problem during OpenAI’s gpt-oss red-teaming challenge. For the experiments later published as In AI Sweet Harmony, I initially used OpenRouter with DeepInfra and Together AI, and also AWS Bedrock. The results were too inconsistent for the research I wanted to do, so I eventually rented H100s through vast.ai and ran gpt-oss-20b myself.
Even then, the inference stack mattered. With otherwise identical experimental parameters and seeds, I measured differences of roughly 5–10 percentage points in how often the model refused comparable prompts across inference stacks. I consequently documented the inference environment as part of the experiment rather than treating the model name alone as sufficient.
For research, I would therefore be reluctant to use automatic provider routing unless provider variation is itself part of the experiment.
Pinning providers - for data protection also
Informed by these benchmark underperformances and sometimes wildy diverging cache hit rates, I take a similar approach in MyChatty: models are always pinned to particular OpenRouter providers instead of simply accepting whatever endpoint the router selects.
Consistency is one reason, data protection is another. I distinguish between US- and China-based providers because choosing an inference provider also determines where data is being processed.
Conclusion
OpenRouter remains useful to me as a common API and catalogue of models and providers. Also, I like keeping just one prepaid wallet, rather than signing up - and funding wallets - with several inference providers. I just would not treat the providers behind a model as universally equivalent - or interchangeable.