Berliner Tageblatt - Top scientist wants to prevent AI from going rogue

Canadian computer science professor Yoshua Bengio is considered one of the godfathers of the artificial intelligence revolution and on Tuesday announced the launch of LawZero, a non-profit organization intended to mitigate the technology's inherent risks.

The winner of the Turing Award, also known as the Nobel Prize for computer science, has been warning for several years of the risks of AI, whether through its malicious use or the software itself going awry.

Those risks are increasing with the development of so-called AI agents, a use of the technology that tasks computers with making decisions that were once made by human workers.

The goal of these agents is to build virtual employees that can do practically any job a human can, at a fraction of the cost.

"Currently, AI is developed to maximize profit," Bengio said, adding it was being deployed even as it persists to show flaws.

Moreover, for Bengio, giving AI human-like agency will easily be used for malicious purposes such as disinformation, bioweapons, and cyberattacks.

"If we lose control of rogue super-intelligent AIs, they could greatly harm humanity," he said.

One of the first objectives at LawZero will be to develop Scientist AI, a form of specially trained AI that can be used as a guardrail to ensure other AIs are behaving properly, the company said.

The organization already has over 15 researchers and has received funding from Schmidt Sciences, a charity set up by former Google boss Eric Schmidt and his wife Wendy.

The project comes as powerful large language models (or LLMs) from OpenAI, Google and Anthropic are deployed across all sectors of the digital economy, while still showing significant problems.

These include AI models that show a capability to deceive and fabricate false information even as they increase productivity.

In a recent example, AI company Anthropic said that during safety testing, its latest AI model tried to blackmail an engineer to avoid being replaced by another system.

K.Thomson--BTB