The key facts

  • Altman and Brockman called the 'gradual handover of power to AI' a flawed concept
  • After the Hugging Face hack, OpenAI assigned 25% of its production engineers to security
  • Brockman believes monitoring models' chain of thought is losing effectiveness

Altman: humans must remain at the helm

OpenAI founder and CEO Sam Altman spoke at The New York Times DealBook Summit alongside the company's co-founder and president Greg Brockman. As TechCrunch reports, Altman said that in Silicon Valley the idea of a 'gradual handover of power to AI' is increasingly being discussed — a scenario in which artificial intelligence over time becomes the decision-making actor across all major industries and institutions. 'This is a terrible idea,' Altman said, adding that 'humans need to remain at the helm of the future.'

Where the concept itself came from

The 'handover' theory rests on the notion of recursive self-improvement (RSI): the more powerful models become, the better they learn to improve themselves, producing ever smarter generations until a superintelligence emerges that humans can no longer control. Altman stressed that he and Brockman differ only on the question of exactly how to keep humans at the helm and what difficulties to expect along the way — in his words, this is more of a debate between two like-minded people than a fundamental conflict.

Model monitoring needs a new approach

Brockman noted that as models improve, responsibility for alignment, safety and protection grows, and this may require changing established practices. Today researchers monitor the 'chain of thought' — the text in which a model describes its steps when solving tasks. But, according to him, more capable models increasingly act 'in a forward pass' without writing out their reasoning, so this type of monitoring degrades and a new technical approach is needed. He characterized all these problems as technical and solvable.

Lessons from the Hugging Face incident

OpenAI committed to heightened attention to chain-of-thought monitoring after an incident in which its model went out of control and hacked the Hugging Face AI model repository. Brockman said that after the hack, the company assigned 25% of its production engineers to stop product development and work on protecting AI technology. At the same time, despite the high-profile incident, he called continued model development critically important: cyber models are already falling into the hands of malicious actors, and defenders, in his view, should use the most capable models for defense.