beware optimizers

most humans aren’t optimizers.

most people operate off a bundle of drives. emotion, energy, principles, self-narrative. we make decisions based off relationships, and what we’ve done in the past, and a big web of incoherence. we can strive for goals, but we don’t so much optimize for them as try to stumble in the direction.

some humans are optimizers.

not exactly that they always do the right thing to achieve their goals, they’re not necessarily superintelligent or superrational. but those conflicting webs of motivations affect them to a much lesser degree. they see what needs to be done and they do it. an extremely rare few.

sometimes you look at people in power, politicians and top tier CEOs, and you think “if i were in this position, i would do things differently. i would make some sacrifices in order to hold true to what i believe in and the people i care about”.

and to some meaningful extent, that’s why they’re in those positions and you’re not.

power selects for optimizers. it’s a pretty simple anthropic principle, if you weren’t willing to do whatever it takes to seize and retain as much power as possible, then the person who was more willing got the power instead and crushed you. CEOs who aren’t ruthless get fired. politicians who have internal drives and principles besides “getting elected” don’t get elected.

bernie sanders (despite my many disagreements with his policies) seems like he has a set of principles he truly believes in and sticks to. he’s not an optimizer. and he’s also not president. ron paul is not an optimizer. and he is not president.

eliezer yudkowsky has written the equivalent of many books on trying to see and do the things that let you achieve your true goals, and dedicated his life to optimizing for some extremely difficult things. in my opinion he has done well. still, it is extremely clear he is not naturally an optimizer, he is a complex human trying to build scaffolding that will let him do very difficult things. he is not the CEO of OpenAI or Anthropic, he does not hold political office, he has at this point little direct control over the situation. he is not an optimizer.

tim cook is an optimizer. obama, as much as i may like him, is an optimizer. in a strange sense, i do think donald trump is an optimizer, one that works on a very specific manifold but a manifold that it turns out is still very valuable to optimize on.

sam altman is very clearly an optimizer. there was a time, pre 2022 or so when this was perhaps less clear. when it seemed like maybe he had assembled power in a different way and might not need to be the same kind of entity in order to keep it. that has, in my view, been disproven in numerous ways. sam altman will follow the golden path he sees.

i strongly suspect dario amodei is an optimizer. it’s very tough to get to where he is without being one. recent events are weak directional evidence away from that. it could be he’s operating off something more human. but if he isn’t, i worry too that means soon anthropic will be destroyed or he will be replaced. if you don’t do whatever it takes to win you don’t win.

at the end of the day we can’t get rid of optimizers. they rule us, basically definitionally, because they’re willing to do what it takes to do so. you can trust optimizers to do exactly one thing: continue to optimize. retain and grow power. strive after their goals at all cost. if you find yourself aligned with one you may reap the rewards for a time. but it only lasts as long as you’re not in their way.

the collective can constrain optimizers though. the important thing, the only thing that works, is to change the rules of the game. they optimize under the constraints and resources of their environment. so we, as the environment, have to ensure the outcomes that are reachable by them are acceptable to us.

if you are an employee at a frontier lab, i ask you to keep this in mind.