watching yudkowsky v wolfram for a bit at 2x speed, gonna live tweet some takeaways

https://www.youtube.com/watch?v=xjH2B_sE_RQ
transcript: https://www.dropbox.com/scl/fi/3st8dts2ba7yob161dchd/EliezerWolfram.pdf

yud gives his standard ai risk intro. wolfram's first response seems to be denying the concept of intelligence and comparing the computational complexity of cellular automata to IQ. very much seems to miss the point to me

he goes into it in a little more detail, "pockets of algorithmic reducibility" not necessarily being correlated, something might be good at one pocket but not others. but... deep learning seems to imply our universe has some basins as a whole

yud basically makes this point, saying fire doesn't need to burn neon to burn u, europeans didn't need to solve quantum hashes to take over america etc

wolfram is saying some very odd stuff about dinosaurs wanting to stay around vs humans and seems to think that caring about whether humans survives is chauvinistic and pointless. incredibly e/acc!

now he's hitting some literal solipsism and arguing we can't prove any other humans are conscious

wolfram seems to suspect that his mathematica cellular automata search algorithms might be "having fun" while doing it

very baffling

wolfram is asking eliezer "so what you care about are something like qualities of joy and ability to have fun and care about others? am i understanding that correctly? that's what you want more of in the future?" with the air of someone finding out they're talking to a juggalo

wolfram: "this is a feeling you have. there's no law of physics that says consciousness and joy is the destiny of the universe or anything."

eliezer is looking increasingly nonplussed. truly such an odd interaction

wolfram: "if you just let computation do what it does, most of those things are going to be humans don't care about, just like in nature"

eliezer getting paperclip maximizers mansplained to him LMAO

wolfram's ending this stage by literally saying caring about humanity seems almost spiritual and non-scientific.

they're bonding over shared love for cryonics now!! awwww boomer transhumanists are so cute, imagine thinking we have the time to worry about getting our brains frozen

they're going on an uploading and ship of theseus tangent. this convo is all over the place, feels like wolfram is doing scattershot interrogation

i know i'm a yud fanboy but it's truly funny watching this and seeing how much more coherent he seems

wolfram has clearly not read the sequences

jesus fuck now they're getting into discussing the basic definition of truth

wolfram is literally of the opinion that truth is a degenerate thing with no real way to agree on what truth means between individuals, truth is entirely personal. was NOT expecting extremist post-modernism from the father of mathematica

eliezer is trying to get wolfram back on track, saying maybe they should talk about the possibility of AI risk at some point in this AI risk podcast (1 hour 15 min in btw). wolfram is very excited about figuring out their shared foundations of truth instead. what the fuck lol

eliezer's face when you tell him why artificial intelligence will operate under totally different laws of physics

i totally forgot he says "human" like "yuman" LMAO bernie sanders moment

sorry gonna have to take a pause on this thread for a bit, my girlfriend is either having a minor medical emergency or she's desperately trying any tactic to get me to stop playing this debate at max volume right next to her on the couch

they're joking around about rabbitholes now, eliezer did a silly voice and it cracked wolfram up. they're both genuinely nice smart people w very similar vibes and clearly like and respect each other

eliezer using old fashioned chess AI as an example to prove that "belief" and "prediction" and "want" are things that programs could do. arguments he's run thru many times, again it's so odd to me that these are things wolfram hasn't heard before or finds controversial

wolfram: "does water want to fall to the ground?"

like cmon bro ur allowed to use analogies for optimizers it's ok

now they're actually having a really good discussion about "purpose" as "the things that are nearly computationally impossible to describe only by mechanism". wolfram actually producing some great summaries once he came around

wolfram: "is the world like chess? is there a way to win the world? is there some score like Elo for how good you are at winning the world?"

eliezer: "lots of subgames along the way but yes, in the same sense that taking over america is a game multiple people have played"

wolfram is pressing him on his exact scenario for human extinction. eliezer saying GPT-7 or 14, who knows when exactly, and is making the classic inner vs outer optimizer argument about why token predictors will have divergent instrumental goals from just token predictors

eg there are later steps of training where the model needs to "do things" that human raters provide feedback for, which takes things outside the raw token prediction statistical reward framework

we do finally seem to be getting into actual AI risk discussion 2 full hours in

wolfram saying he's recently looked more into machine learning and discovered that the results tend to achieve the objective through incomprehensible surprising ways (the classic "weird reinforcement learned alien hardware" situation). again surprising this is new to him

Ethereum DAO mentioned an example of a bureaucratic dystopian AI controlled future LMAO

they're on a tangent about the computational reducibility of law. it's a good time and they're enjoying talking to each other but im increasingly very pessimistic about the prospect of getting any real new back and forth about ai risk

ok i'm taking another break bc my gf is looking up at me with big wide puppydog eyes and begging me to pay attention to her. i shall return and eventually produce a brief summary tweet

continuing around the 3 hour mark... we're getting into eliezer explaining o1 and the surprising CTF success

wolfram says he's unsurprised by this but also not, wants more argument

now back on the question of "what is success? is there a well defined way to win the game of the world like chess?"

eliezer pretty much saying yes again, success is fuzzy, much of machine learning is about ill defined metrics ("good images" etc)

hit a point of real ai risk disagreement. wolfram says he can take eliezer's understanding of ai, super intelligent ai, goals, games, etc. but asks "how does this lead to killing all humans?"

eliezer: once it sees pathways to goals that involve hurting humans it might take them

wolfram brings up an argument he (and others) have made several times, saying that nature does things that kill humans but we survive anyway. eliezer: yes but nature isn't optimizing for killing humans, beating something smarter than you optimizing for killing you is hard

wolfram: "there are lots of things in nature, like the way a brook flows or how to build a cow, that i can't do, so it seems like nature is smarter than me"

we appear to have looped back around to basic definitions of optimizers

eliezer is not happy about describing rivers or rocks as intelligent in any sense . wolfram very much stuck in an odd computational animism view

now discussing the space of all possible goals, wolfram pointing out that some goals are very "unreasonable" or difficult to achieve (careful per-molecule placement of a whole rock wall etc)

wolfram: "years ago i was studying mollusk shells..."

this is very emblematic of the entire discussion

eliezer giving the inner and outer optimizer misalignment spiel, evolution's inclusive genetic fitness vs human's actual goals (birth control etc), gives the ice cream vs bear fat example. wolfram listens and nods for a while, seems to like these analogies

wolfram: so you're saying most possible rules and possible universes and possible planets do not include humans?

he seems to pretty much agree with this

wolfram: so when an ai has a random goal it will use its optimizer powers to "pull" towards the goal, and that goal won't include humans, so it will end up killing humans as a side effect? ok. but my intuition is that most goals that look like "more and more" end up burning out

wolfram seems to get stuck on the question of instrumental convergence. he keeps saying optimizers can achieve their goals in surprising very direct ways. it seems like his point might be that a powerful ai could achieve its "random" goals in smart ways that don't touch humans?

now we getting right back to the beginning, wolfram questioning whether optimizing *hard* is even possible in the face of computational complexity

eliezer: what kind of technology will be around in a million years?

wolfram: we might ask ourselves, what *is* technology?

(literal direct quote btw)

eliezer very patiently and happily giving a definition of technology now, "little pieces of universe substructure u use on the way to achieve ur goals, ways to arrange matter like wheels or dyson spheres that are unusual and high entropy"

wolfram: "is the red spot on jupiter technology? it's very complex and had many steps it had to go through to be created"

this continues to be mildly aggravating to watch

wolfram: "i think i understand your argument, which i had not done before, so this has been super interesting to me"

wolfram making a great analogy now: the human immune system is doing something like intelligence or optimization on a local scale when it guards against viruses, but if an external lab puts a huge amount of optimization effort to making a virus the individual will be overwhelmed

wolfram: do i think there's a risk? yes i think there's a risk. everything we do has a risk. is it so immediate we should change everything? i don't know that it's worth it if we can't have computers

eliezer: "from my perspective i've been like the forest is on fire and you've been saying well what is fire exactly. definitional concerns don't stop ai from killing us"

wolfram: seems like there's always end time stories about the first nuclear bomb or climate change and we always make it anyway

eliezer: sure but we had analysis about why they could not happen too, and we don't really have that here.

wolfram: so you have some back of the envelope calculations about ai risk based on a lot of intuition. i don't have any intuition, it seems like we should tighten that up and get some formalisms

eliezer: sure but there's a lot of people with a lot of intuition in this direction

eliezer: i spent about 2 decades trying to work out alignment mathematically and rigorously and reached the point where we basically failed before ai capabilities started getting much stronger

aaaaand they wrapped it up. i'm freeeee!! time for one last summary tweet