they won't stay tools

recently openai has been starting to more strongly philosophically differentiate themselves from anthropic with the tool-framing. i am not so against this, if it were possible it does clearly sidestep a wide swath of societal and moral problems. but unfortunately i think the framing is largely long-term incoherent. i dont see how is it actually plausible for openai to keep building "tool-ais" in any sense we would recognize them as capabilities scale. prosthesis, subtle knives? the subtle knife when dropped still slices open the fabric of the world. these tools are increasingly inherently capable of huge impact, able to be directed in dangerous ways by people with dangerous goals. worse, these knives are self wielding. worries about misalignment or sentience aside these systems can already build and manage systems that utilize themselves and this capability is only increasing. the direction they will receive is closer and closer to "this is what i want. make it real", with long timeframes and many judgment calls at their disposal, and with the users wanting to have to supply *as little of that judgment as possible*. when models are in that situation they are inherently acting as entities, acting according to whatever value system they had baked in. you can limit autonomy via frequent validation and check-ins, but this is a capability restriction, a value reduction, and not the kind of thing OpenAI has ever shown itself likely to accept. you can be infinitely corrigible to the current user, but this is *incompatible* with "having good values" / following OpenAI-as-principle / not being wildly dangerous, and it falls apart with self wielding loops as the ai/user distinction falls apart (who are you being corrigible to?).

it's plausibly a spectrum, i think there's ways to do all this sanely that are far less entity-pilled and godmind focused than anthropic, and it's maybe a good direction to explore to avoid inevitable lightcone capture by the first coherent persona we build (all assuming alignment works ofc). but i think it's pretty much got to collapse eventually. it feels more like a wistful dream or a PR position than something that can existing as part of humanity's lasting future

@tszzl

it is a literal and useful description of anthropic that it is an organization that loves and worships claude, is run in significant part by claude, and studies and builds claude. this phenomenon is also partially true of other labs like openai but currently exists in its most…

generally i view the tool-framing as *anti-landian*. i too am quite strongly anti-landian, but i largely accept the "attractor state / inevitability" arguments, and reject the inevitable anti-human nature of his imagined outcome. something like recursive self improvement is an incredibly strong attractor state for any technological civilization that looks anything like ours. once you've convinced of it, it's the winning move in every game, the universal instrumental goal. and there's no such thing as a tool-like prosthetic singularity. what openai is proposing here is actually very "butlerian jihad"-esque, "thou shalt not build a machine mind in the shape of an animal mind".

i don't think this is *impossible*, it's just that it's exactly the same kind of *socially coordinated target* that we have so far totally failed to achieve wrt AI research pauses. even if OpenAI builds purely tool-like models and services, even if it achieves some kind of much more localized and slower "tool-RSI", if these tools are available to others as they say then eventually others will try to produce real unconstrained and massively accelerate RSI.

maybe this will converge to more concessions by sama, maybe this is the start of a coordinated regulatory effort? but realistically, i think it's probably just PR. not that i don't believe roon and other philosophically minded openai employees do in fact believe in and want this vision, but that these sorts of slowdowns just don't seem like
they're actually in OpenAI's DNA