alien minds
let's go thru these one by one
llm mindspace
- the mindspace is indeed limited, but 1. the mindspace of token predictors trained on human text is not at all the same as actual human minds, in ways we have very little understanding of. these are things that can simulate and switch between arbitrary persona. they are in fact incredibly alien.
llm values
- it does seem true that llms can ~understand human values, although they remain as easily confused as many humans. unclear to me that this was ever a central point or assertion of ai risk arguments, so much as a potential challenge that was discussed and turns out to not matter.
recursive self-improvement
- seeing the no free lunch theorem mentioned in an argument about ai is a lot like seeing someone try to use godel's incompleteness theorem to talk about consciousness. 99% chance they do not in fact understand the mathematical content but it feels good to hand wave. no free lunch says that u can't find an optimization algo that works across *literally all possible mathematical distributions*, including noise and degenerate ass constructions. it absolutely allows for alpha in all kinds of subspaces, for example "distributions that we actually see in our physical universe". hence why neural networks work well in the first place. we've literally just seen deepseek hit massive multiples on cost / quality ratio due to... research improvements. sure current paradigms are extremely compute and data heavy, but we absolutely do not have mathematical guarantees that will remain true. and in fact multiple labs are investing massively in making LLMs useful for ML research to speed up the process and... recursively self improve.
now i will say there is some update here, since ~20 years ago it wasn't clear what kind of hardware or compute ai would require . we're definitely not getting a basement situation, but that absolutely still leaves the data center situation.
faking alignment
- idk what the nonsense about regularization is here bc uhhh unfortunately it's incredibly well established that what humans think of as "perverse" is not what u get out of training models. basic RL will get u shit like hacking the reward function, and anthropic + others release a new paper showing literal alignment faking and cheating in current LLMs like once a month. absolutely insane argument, this is one of yud's most well established wins.
incorrigibility
-once again: literally shown to happen on current systems https://t.co/dXC1EA91g3
alignment of smarter models
- the claim that alignment is easy and we've already mostly done it is just transparently false, jailbreaks happen constantly and papers like the above get released once a month. models are already autonomously editing their eval scripts to game benchmarks, which is precisely the kind of thing that shows how smarter models could get harder to align.
conferences
- miri was trying to actually figure out alignment, not delay ai at all. at the time people pretty much thought we still had decades. sure, they didn't manage it, but it's not like anyone else has either.
anyway yeah overall the world now doesn't look quite like lesswrong expected in 2011. but it looks a hell of a lot closer to what lesswrong expected than what literally anyone else expected. and from my pov none of these points does much of anything to actually argue against ai risk.
@RokoMijicThe Less Wrong/Singularity/AI Risk movement started in the 2000s by Yudkowsky and others, which I was an early adherent to, is wrong about all of its core claims around AI risk. It's important to recognize this and appropriately downgrade the credence we give to such claims…