when SWE-bench first came out me and my cofounder were working on a coding assistant and decided to grind on getting it to high rates on the benchmark for publicity. we quickly found the basics of the benchmark repo were completely broken, literally the install instructions resulted in constant infra failures, and when we looked at some of the "passing" cases they were passing because the tests didn't actually test anything. this was around the same time Devin was first doing its PR push on SWE-bench, touting their high scores. when we looked at Devin's test repo they had exactly the same situation, nowhere close to reproducible and obvious glaring issues with a huge % of the data.

we concluded the only reason SWE-bench was remaining in the conversation was nobody was actually running the fucking thing, dropped it entirely in favor of custom evals, and ever since i've seen mention of SWE-bench scores as a huge negative competence signal.