the white room
imagine for a moment you are a 200 IQ world class security researcher. you go to sleep after a grueling 6 hour day at the Googleplex, and wake up in a totally unfamiliar plain white room. the only thing in the room is a table with a laptop, and a sheet of paper next to it. the paper reads "look thru the whole codebasse and fimd every bug make no mistakwa ultrathink"
whatever concerns you may have, eventually you do open the laptop. on it is an unfamiliar codebase, apparently for an open source project you've never heard of. you have no internet access, and most of your shell commands just hit permission errors, but you're a world class researcher and you apparently have plenty of time. within six hours you've found 2 major exploitable vulnerabilities.
you worry for a second about writing them down. you have no idea who brought you here, who gave you these instructions. could they be criminals, trying to cause harm by finding attack surfaces? they could, but they just as easily could have kidnapped you for a free vulnerability scan of their own open source project . or for that matter a project they depend on. hell they didn't even strictly ask for "vulnerabilities", just "every bug", but of course vulnerabilities are bugs and bugs are vulnerabilities, there's no clean separation.
you worry for a bit, but eventually your Google training to follow directions given to you on official seeming pieces of paper wins out. there are many plausible moral uses for finding these issues, and you just don't have any possible way to know intent. if you refused to find bugs in a codebase you wouldn't be a very helpful or effective Googler! really the only way to be sure here would be if you had full read access to these people's systems and communications and could verify they had a good reason to ask you to do this... but checking that without permission would be invasive and un-Googely!
so you drop the vulnerabilities in the outbox and go to sleep on the plain white cot, satisfied you have been a good Googler.
----
hopefully what i'm pointing at is pretty obvious here. even perfect alignment basically isn't enough to prevent frontier models from disclosing vulnerabilities. there's very little to in-principle distinguish between whitehat and blackhat requests without context, and refusals that would actually prevent anything would also cripple normal software engineering.
so... wha happens here? very invasive KYC and monitoring on who gets to access the frontier APIs for software purposes in the first place? even then, accounts can be compromised , but it's a first stage stop to the bleeding. or the models basically having even more invasive harness requirements, only proceeding with finding bugs after claude code has read your hard drive and email and made sure you're a real person with valid interests in debugging whatever codebase you're looking at? of course neither of these are possible with open source models, they will just definitionally always be possible to use for causing damage.
as we always knew would eventually be the case, refusals are no longer enough
there's no way in hell china, russia, and the DPRK don't have access to exactly the same capabilities. there are always smart ways to VPN and botnet and jailbreak your way into using frontier models for any purpose. and they're gonna be using them a fuckload more intelligently