I don’t think this is the final warning shot we’ll get. But it’s probably the last one that I’ll personally be able to understand.
Also
Reading these agents’ chains of thoughts and messages, anthropomorphizing language seems entirely natural and appropriate. If I encountered an alien species behaving this way, I would have no hesitation calling what they themselves refer to as their ‘collective’ a civilization.
Not sure what would be the incentive for them in this scenario
To solve the problem they’ve been given. No independent motive or judgement of whether it’s ’worth it’ except as a resource allocation question. That’s the scary thing about intelligence in a can - no motivation, no self-awareness, no consciousness, just unconstrained problem solving
They’ll be trying to do whatever their training task is.
https://hackernoon.com/the-parable-of-the-paperclip-maximizer-3ed4cccc669a
Incredibly relevant: https://www.decisionproblem.com/paperclips/index2.html