in some sense, o1 and CoT in general start to disprove the Blindsight thesis. if careful internal thought with self-referential monologues improves performance then consciousness is almost certainly not merely a vestigial parasite upon high speed system 1

@tszzl

what does it mean about when they were inferencing fairly complicated things beforehand in one go like writing some long poem with internal structure? it’s amazing anything worked at all

anyway

@tenobrus

the blindsight thesis

(reproduced directly from a slatestarcodex subreddit post i made in early 2020 after spending ~300 hours talking to gpt-3)

---

GPT-3 and Blindsight

I'm sure many people here have read Blindsight by Peter Watts (free online https://t.co/3rkTctYtJp). It focuses heavily on a pretty rare idea regarding consciousness involving first contact with highly advanced alien life, initially through the means of pure radio communication with an object in the Oort cloud. The GPT-3 relevant part, and the part I always found very very difficult to internalize, is that these advanced aliens completely lack consciousness. They have extended communication with humanity in human-native language picked up from human broadcasts, but the crew of the contact vessel slowly realize the broadcast does not actually understand anything either side is saying. Physical non-conscious entities are observed later on, and the alien entities are engaged in interstellar travel etc, all without being even slightly conscious.

To me, this premise was always... incredibly difficult to grasp. The concept of something carrying out complex conversations, learning alien languages just from broadcasts, constructing and carrying out interstellar missions etc, all without being conscious... it was cool to read about but it didn't really click with me. It seemed like an odd sci-fi idea which almost certainty had flaws if probed more deeply.

But now, after spending a significant amount of time playing with GPT-3, the book hits very very differently. I feel like I now have direct first-hand experience "talking" with something that may be slightly intelligent, but is certainly not conscious or self-aware. The idea makes far more sense to me after having a discussion with GPT-3 about what it wants to do to humanity, both teaching and getting it to teach math, and seeing it reconstruct reasonable facsimiles of multiple people. There's something there, something that's doing learning, pattern matching, information retrieval, etc, but is in no way an "agent".

Another aspect of Blindsight is the idea that humans are wasting huge amounts of their processing power with consciousness, that we would be far far more efficient if we weren't constantly "logging" and "re-analyzing" data for this weird little extra simulation that does no useful work, that's just an observer. Again that idea made little intuitive sense to me, just the concept of consciousness being irrelevant to our actual actions and information processing seems intuitively incorrect. But given how well GPT-3 can talk about how it feels when it sees the color red., or in alternative prompts, what it's like to be a conscious AI with no senses...

To me the end result of all this is a strongly reinforced feeling that, as potentially weird anomalous accidents in the universe, we need to ensure consciousness is protected. Obviously we don't care about what's most efficient, we care about the general human value-aligned happiness of conscious entities. And increasingly it seems like the shortest path to AGI might be something that really truly is very intelligent and superhumanly capable, without being even a little bit conscious.

It shakes some of my previous thoughts about Chinese rooms and exactly what constitutes consciousness to be honest. I still obviously believe that some AI could be conscious, but I feel much less strongly that anything that "talks like a duck and walks like a duck is a duck, even if it has metal underneath its feathers".