There is a fear of people that AI can get more intelligent than people and can take actions against them. Some people say that for a decision to make harm humans need to be conscious. So, if we can be sure that “AI” is not conscious, there is no real danger.
Sir Roger Penrose (Nobel) and Gödel’s Theorem
Penrose explains that Gödel’s theorem demonstrates that „there are things that understanding transcends the use of rules.” This, he argues, highlights a critical difference between computation and true understanding.
He explains the theorem: „You construct a statement which says…I am not provable by those rules.” The implication is that a system following rules cannot inherently understand those rules in a way that allows for transcending them. This inability to transcend rules, according to Penrose, is crucial in defining consciousness.
Understanding vs. Computation
Penrose draws a distinction between simply using rules (computability) and understanding why rules work (consciousness). He argues that „understanding them requires being conscious of them.” True understanding, he asserts, necessitates consciousness, a capacity he believes current AI lacks.
AI’s Lack of Understanding
Penrose stresses that AI, however powerful, does not understand what it’s doing. He states: „AI doesn’t know what it’s doing. It speaks from experiences, if you like.” He attributes the misconception surrounding AI to the immense power of modern computers, leading to a misapprehension of their capabilities.
Source: https://www.youtube.com/watch?v=e9484gNpFF8
If it can act…
Some people think that there is a philosophical explanation of consciousness and there is a real experience with AI. Professor Jeffrey Hinton, the „Godfather of AI,” says that AI can take over the world. How?
AI agents can be atonomous
Professor Hinton explains how AI agents, capable of independent actions and subgoal creation, could potentially seize control. „As soon as you have agents, you get a much greater chance of them taking over.” He emphasizes the inherent drive for AI to gain more control to better achieve its goals.
Source: https://www.youtube.com/watch?v=vxkBE23zDmQ
I am optimistic
Who am I to tell who is right? However, I can say what I think as everyone can have an opinion. I think it is a real danger that agents will set up subgoals that are dangerous, but at the same time I think they are not conscious. I also think that being conscious you must be a person and that thinking is not only a mental movement – but this is beliefs, not science.
What if they could… imagine
A more scientific reason is that “the Godfather of AI” often says the words “imagine if”. We can imagine anything. Many things that will never happen. Imagining is not a fact, but it doesn’t mean it can not become a fact. I think Professor Hinton warns us, and it is very useful.
All I want to say here is that an unconscious AI can do harm, but I think people could prepare for it. I think that stopping developing LLMs for a year would really help. If there is one thing that could kill us all is the headless running for returns on invested money. Is it a new challenge?