AI researchers warn of existential risk even as firms chase profit

During a Goldman Sachs technology summit in San Francisco, OpenAI CFO Sarah Friar announced that the firm’s biggest models can now train smaller ones, illustrating recursive self-improvement. This development has alarmed parts of the AI community; Anthropic alignment lead Evan Hubinger posted that he believes there is over a 10% probability AI could eradicate humanity within ten years. Other staff at Anthropic and OpenAI echoed the concern, while Altimeter Capital investor Brad Gerstner called the warning "fucking nonsense."

Analysts attribute continued AI work to strong commercial incentives, a conviction that their labs are uniquely qualified to keep the technology safe, the ability to attract researchers through safety-focused missions, and a growing sense among engineers that they have little control over AI’s direction. Sam Altman recently stressed the importance of agency in shaping AI’s future. The episode highlights the tension between lucrative AI expansion and existential risk debates.