Agentic harnesses for vulnerability discovery
Pablo Ruiz shared his CodeCome vulnerability discovery harness last week:
In short, this is a 'mini' harness for orchestrates AI agents to inspect source code, generate vulnerability hypotheses, validate them in sandboxed environments where possible, and produce evidence that humans can review. The goal is not magic “AI finds all bugs”, but a practical playground to explore how different models and agentic workflows behave on real AppSec tasks.
It’s built on OpenCode, so benefits from “Login with Codex” or OpenRouter integration. Pablo notes that normally any SoTA model is good enough, but Gemini 3.1 “finds the solution quicker”.
This week, Cloudflare published a report on their harness, with Anthropic’s Mythos model plugged in:
They confirm Paolo’s assessment that other models can be plugged in for some of the same work, but Mythos hits different:
Where [frontier models through the same harness] fell short was at the point of stitching the pieces together. A model would identify an interesting bug, write a thoughtful description of why it mattered, and then stop, leaving the actual chain unfinished and the question of exploitability open. What changed with Mythos Preview is that a model can now take those low-severity bugs (which would traditionally sit invisible in a backlog) and chain them into a single, more severe exploit.
Interesting and consistent with my red-teaming results for GPT-OSS, rephrasing prompts pushes guardrails:
Despite this, the model organically pushes back on certain requests - much like the cyber capabilities that made it useful for vulnerability hunting, the model has its own emergent guardrails that sometimes cause it to push back on legitimate security research requests. But as we found, these organic refusals aren’t consistent - the same task, framed differently or presented in a different context, could produce completely different outcomes
Someone from Anthropic attended our workshop back then, and this could be another reason why they are not making it public - it could be raw, and possibly pose a safety concern not just in the cyber realm.
Update 2022-05-22: Daniel Stenberg, Maintainer of the curl web access tool, reports the highest number of confirmed vulnerability reports since when they got their first security audit in 2016 - and there are three reports left in the queue, with more coming in. AI tools are the reason he says, and he sees Mythos as a distraction:
The simple reason is: the (AI powered) tools are this good now. And people use these tools against curl source code.They find lots of new problems no one detected before. And none of these new ones used Mythos. Focusing on Mythos is a distraction - there are plenty of good models, and people who can figure out how to get those models and tools to find things.