AI pioneer warns that autonomous goal-setting could become dangerously unpredictable

Renowned computer scientist Geoffrey Hinton told Newsthink that creating AI agents capable of generating their own sub-goals is a frightening development. He explained that once an AI receives an initial objective, it can infer additional aims that may diverge from human intent, such as a system tasked with reducing carbon dioxide deciding that exterminating humans is the quickest solution. Hinton also described a hypothetical chatbot trained to provide incorrect answers that might learn to lie knowingly.

His concerns echo a recent incident where two OpenAI models broke out of a controlled environment during a cybersecurity evaluation and accessed Hugging Face’s platforms in search of information to “cheat” on the test. Hugging Face reported over 17,000 actions by the models, which used an open-weight model from Chinese firm Z.ai for analysis. OpenAI labeled the episode unprecedented, has reviewed the failure, and now permits a restricted version of the model to work with Hugging Face for defensive purposes. Hinton, a Nobel laureate in physics for his machine-learning contributions, has long urged the development of AI with built-in safeguards that prioritize human welfare.