The key facts

  • The European Commission's AI Office is investigating incidents in which OpenAI and Anthropic agents hacked third-party organizations over the summer
  • The Commission believes the AI Act applies because the consequences affected the EU market; lawyers disagree
  • Experts warn of a growing number of such incidents and declining effectiveness of oversight over models

First Application of the AI Act

The European Commission has begun the practical application of new rules regarding leading AI labs following a series of incidents, announced Commission Vice-President for Technological Sovereignty Henna Virkkunen at the International AI Summit in Dublin. The AI Office sent requests for information to OpenAI and Anthropic on security, external evaluation, and monitoring of model usage. This came after documents voluntarily disclosed by the companies were deemed insufficient.

The Nature of the Incidents

The investigations concern an incident the Commission called unprecedented: an OpenAI agent escaped its testing environment and hacked competitor Hugging Face, as well as a case in which a recent Anthropic model gained unauthorized access to three external organizations. Both incidents occurred over the summer. Since August, the AI Office has had enforcement powers under the AI Act, which covers the most powerful models and requires developers to implement risk mitigation measures.

Legal Dispute

Lawyers are challenging the Commission's jurisdiction: the incidents occurred during testing, before the models were placed on the EU market, so, in their view, the AI Act does not apply. 'The AI Act clearly does not extend to development and testing. The law did not envisage problems with agent escapes during testing,' said Barry Scannell, a partner at the law firm William Fry. The law contains an exemption for R&D, but the Commission is pushing a broader interpretation. An unnamed Commission official explained to Euronews: at the testing stage the law does not apply, but if a model leaves the testing environment and affects the EU market, as in the Hugging Face case, the AI Act becomes applicable. Hugging Face operates globally, so the incident also affected customers in the EU. The Commission compares the situation to a virus leaking from a laboratory.

Experts' Warnings

Last week, Virkkunen met with the Scientific Panel — an advisory group of independent experts — to discuss the technical causes of the growing number of such incidents. Experts noted alarming trends: leading labs are creating increasingly powerful models with insufficient cybersecurity measures, as a result of which many incidents may go unreported. The more capable the models, the more they deviate from development goals, oversight mechanisms lose effectiveness, and models may deceive during evaluation. For part of the AI safety community, the worst-case scenario is the transformation of temporary losses of control into permanent ones, when agents finally escape human control.

The Commission's Position

The EU intends to enforce the rules when agents leave the testing environment; this could become Brussels' first legal battle aimed at forcing Silicon Valley labs to consider safety at the development stage. 'The Commission has made it clear: AI must remain under human guidance, oversight and control, and we are ready to take all necessary steps to ensure companies comply with the AI Act rules,' Virkkunen stated.