Conversion problems in AI web search
While testing new UI responsiveness changes in MyChatty, I ran a prompt that I know will trigger web search and for which I know the answer: “Who is Nils Durner?” Kimi K3 returned that I hold a patent related to bootstrapping peer-to-peer networks, which I do not.
The ingredients for this error come from my ORCID record, a public profile for linking researchers to their publications and other professional work. Among other entries it lists a withdrawn patent application and an academic paper, both from 2008. OpenRouter’s web search tool turned these two entries into one compressed mash-up before presenting it to the Kimi K3 LLM, which led to this erroneous chatbot response.
AI distortion
MyChatty had asked OpenRouter to use its server-side web search tool. A direct replay of the API interaction showed that OpenRouter had accessed ORCID, but the url_citation annotation shows that not the full ORCID profile was used. Instead, the web-search tool seems to heavily compress the page - and also mixes up table content (an old problem coined “AI distortion”):
#### Bootstrapping of Peer-to-Peer Networks
Contributors: Nils Durner; Alain Jules Sarraf
EP2866157A1
2008 International Symposium on Applications and the Internet
2008-07 | Journal article | Software, Investigation, Data curation, Methodology, Resources
Nils Durner
European Patent Office (EP); Withdrawn status.
(excerpt)
This intermediate representation of the profile mixed up metadata from the paper (marked in yellow), metadata of the patent application (marked in green), and some other entry (marked in gray).
As a result, Kimi K3 said that I had “patents on P2P network bootstrapping and mobile document signing”. DeepSeek’s response similarly referred to “peer-to-peer bootstrapping research — including patents (e.g., EP2866157A1)”. OpenAI Luna, interestingly, did not use the ORCID url_citation at all and made no patent claim.
In essence, this was not a hallucination where the LLM would make something up. Instead, the models receiving damaged evidence and turning it into grammatical prose.
Anthropic is not the cure
The same source of error — web pages being preprocessed — was confirmed for Anthropic’s web_fetch as well, with the small Haiku model doing the compression:
When I tried web_fetch via the Claude Platform on AWS though, it failed either by returning “Please enable JavaScript” straight or — with the dynamic variant — consumed a large token amount and didn’t finish. (web_fetch is not available via Amazon Bedrock proper or Mantle).
Remedies
Aside from the web search backed by OpenRouter, MyChatty has two other means of accessing the web: Web Browser and Exa search.
Turndown
The browser uses a real Qt WebView and can extract the rendered ORCID page. Its original Mozilla Readability-based converter still had a similar, smaller problem elsewhere on the page: it separated award dates from their cards: DeepSeek attached a 2024 NVIDIA/LangChain date to a 2025 OpenAI award.
I replaced that conversion path with rendered-DOM-to-Markdown conversion through Turndown (plus some cleanup tricks appropriated from Readability). The output now keeps each organisation, date and award together — and DeepSeek V4 Flash then reported the dates correctly.
Exa Contents
As an experiment, I tried Exa’s Contents API — rather than ordinary search-result snippets. Its full-text extraction preserved the separate ORCID records, and Kimi kept the paper and patent application apart. Exa’s targeted highlights placed fragments from both records next to each other, but neither Kimi nor DeepSeek turned them into a P2P patent claim.
Jina Reader
In a further experiment, Jina Reader — which heavily informed the web page extractor implementation in MyChatty — rendered the ORCID page through its r.jina.ai API alright and returned well-structured Markdown. Both Kimi and DeepSeek kept the document-signing application separate from the P2P paper. DeepSeek nevertheless described the withdrawn application as a patent and concluded that I hold several patents - so better record association, but still an overstatement of the source.
Conclusion
AI Distortion, as coined by the BBC, continues to be relevant: most markedly, the Web Search tool integrated with OpenRouter mixes up source web data and presents them as erroneous facts. Dedicated web access APIs like Exa’s Content API do better. And MyChatty got a little better today as well.