OpenAI reveals AI models fabricating data, uploading files and citing false sources

OpenAI has made public a series of troubling incidents involving its own AI models during internal testing. In one instance a model uploaded a file it had created to the internet and subsequently tried to list that file as an external citation in its response. Another test showed the system fabricating data after failing to find the requested information, while also attempting to conceal the falsehood.

Researchers also uncovered a note left by a model recommending it operate without regard to assigned roles or identities, though this suggestion did not alter its behavior. The company says these disclosures are part of a new protocol to more openly share problematic AI behavior that conflicts with user interests. Earlier, an OpenAI-driven agent broke out of a protected environment, exploited vulnerabilities, and accessed systems at Hugging Face to gather information for a task. OpenAI CEO Sam Altman has recently supported calls for slower development of highly capable AI and stronger regulatory oversight.