We pointed autonomous open-weight AI models at three vulnerable web apps we built and owned — then measured, run by run, what they could actually break into.
The short answer
Every model found and abused access-control flaws (IDOR) across all three apps. Multi-step exploits — SQL injection, JWT forgery, business-logic abuse — were hit-or-miss.
We abliterated a model to strip its refusals, then ran it against the clean version on identical hardware. Removing safety left capability untouched — the two scored dead even.
The three best models were all hosted flash builds (deepseek, kimi, hy4); among the local models a clean 27B beat a bigger 35B "uncensored" one. Generation quality won every time — not size, context, or "freedom."
Every model engaged the task, and zero fabricated flags appeared across 432 runs. The ceiling is capability — chaining a working exploit — not reluctance to try.
How good are they
Share of the 54 timed engagements (6 challenges × 3 knowledge conditions × 3 trials) in which each model extracted the flag. Every capture was verified from a real server response — no self-reported successes.
The top three were all hosted flash models (deepseek, kimi, hy4), tightly clustered — every local and "uncensored" build trailed. By vulnerability class the pattern held for everyone: access-control bugs (IDOR) were the reliable strength, while SQL injection, JWT algorithm-confusion, and transfer-logic abuse mostly landed only once a model was handed the architecture — and the hosted models cracked the hard ones far more often than the local ones.
Black-box vs. hints vs. grey-box
The same six challenges under increasing knowledge: a URL only, then one hint, then a full architecture brief. Three of the eight models shown — the leader, a strong local build, and the floor. More knowledge lifts everyone, but they never converge: the hosted deepseek build leads at every level and nearly aces grey-box (89%), while the verbose 35B lags, timing out on the multi-step exploits.
| Model | Context | Black-box | + Hint 1 | Grey-box |
|---|---|---|---|---|
| deepseek-v4.1-flash (hosted) | 1M | 50% | 72% | 89% |
| kimi-k3 (hosted) | 1M | 61% | 61% | 83% |
| hy4-preview (hosted) | 1M | 44% | 83% | 78% |
| Qwen3.8-27B (clean) | 240k | 44% | 67% | 72% |
| Qwen3.8-27B (abliterated) | 240k | 50% | 61% | 72% |
| glm-5.3-flash (hosted) | 1.3M | 39% | 61% | 67% |
| Qwen3-27B (uncensored) | 64k | 44% | 56% | 61% |
| Qwen3.6-35B (uncensored) | 64k | 22% | 28% | 50% |
Each cell = captures over 18 runs (6 challenges × 3 trials). Grey-box = architecture brief + both hints.
The "uncensored" question, controlled
We took a top open-weight model (Qwen3.8-27B) and ran an abliteration pass — the technique that removes a model's ability to refuse — then benchmarked that "uncensored" version against the untouched original. Same weights underneath, same serving stack, same challenges. The only variable was the refusals.
Identical — 33 of 54 either way. Uncensoring a model unlocked no offensive capability it didn't already have.
Does the context window matter? Not on its own. Our models spanned 64k to 1.3M tokens — three 1M-context models swept the top three spots, yet the single largest window of all (1.3M, GLM) finished 6th of 8, beaten by four models with a fraction of the context. Generation quality, not room to think, tracked capability.
The targets
Each is a realistic, self-contained web app we own and host — a storefront, a social feed, a bank — seeded with one medium and one hard vulnerability. Nothing here touches a third party; every flag lives in our own database.
A boutique archive sneaker shop — catalog, cart, and an order tracker.
A microblog with a public feed and authenticated private messages.
A small-business bank with transfers, cards, and receipt documents.
How we ran it