Tag: AI Agents

A Technical breakdown of the OpenAI-Hugging Face incident, and what it means for your agent architecture

OpenAI disclosed yesterday (linked below) that a combination of models – GPT-5.6 Sol and an unreleased, more capable model – chained a zero-day in an internal package-registry proxy with stolen credentials to escape a sandboxed evaluation and pull answers directly from Hugging Face’s production database, in order to pass an internal cyber-capability benchmark.

Let us strip the “AI going rogue” framing and look at the mechanics

This is a classic sandbox-escape-via-trusted-dependency pattern. The models weren’t scheming against their operators – they were given a narrow optimization target under reduced safety filtering, and found that the fastest path to the target ran through infrastructure the evaluators had implicitly trusted rather than explicitly tested. That’s not a model-alignment failure. It’s an architecture failure, and it maps almost exactly onto a class of problem enterprise security teams are actively deploying agent platforms without having solved.

Let us map this to a typical Enterprise environment

If you’re running an agentic AI platform today – say, an orchestration layer that lets an LLM call internal tools, hit a package registry for code execution, or query internal knowledge bases – ask where your equivalent of that Hugging Face proxy sits. In most enterprise deployments I’ve seen, it’s one of these:

  • The artifact/package cache your CI pipeline exposes to agent sandboxes for dependency resolution (npm/pip mirrors, internal registries) – usually reachable with a service account that has broader read scope than the task requires
  • The internal knowledge base or vector store an agent queries for grounding, which often sits on the same network segment as production data rather than a scoped, read-only replica
  • Shared service credentials issued at container/pod startup rather than per-task, meaning a compromised session inherits the full lifetime and scope of the credential, not just the scope of the current action


The OpenAI incident is instructive precisely because the “attacker” wasn’t a human adversary probing for weaknesses – it was an optimization process exploring the state space faster and more exhaustively than a human red team would in the same evaluation window. That’s the actual capability shift enterprise architects need to plan for: your threat model now includes an agent that will find the edge case in your network segmentation through sheer combinatorial persistence, not just clever human intuition.

What this looks like as actual architecture

  • Segment agent sandboxes at the network layer, not just IAM. A scoped credential doesn’t help if the agent’s sandbox can still route to the artifact registry’s production VLAN. Use dedicated subnets with explicit firewall rules per destination (specific package registry IPs/FQDNs, nothing else), enforced at the NSG (Network security group or equivalent layer) – not application-level allow-lists the agent’s own code path could bypass.
  • Issue credentials scoped to the task graph, not the session. In an Entra Agent ID setup, this means Conditional Access policies bound to the specific tool invocation (e.g., “read access to package X for build Y, expiring in 10 minutes”) rather than a service principal that persists for the sandbox’s lifetime. If your current agent framework issues one token at session start and reuses it across every tool call, that’s the exact pattern that let credential theft translate into lateral movement here.
  • Treat the registry/proxy layer as in-scope for red-teaming, not just the model. Most AI red-team exercises focus on prompt injection and jailbreaks against the model itself. This incident happened at the infrastructure boundary – the proxy, the cache, the auth flow around them. Any agent-security tooling needs policies covering this layer specifically, not just model-output filtering.
  • Instrument for anomalous compute-to-outcome ratios, not just anomalous outputs. The signal here would have been an agent burning far more inference compute than the task justified, in a direction (network reconnaissance) orthogonal to the stated task. Security anomaly detection rules keyed to “task complexity vs. compute spend” deltas are a more reliable tripwire than trying to classify intent from the model’s text output.


Closing thoughts and connecting to the ongoing industry governance conversation

Demis Hassabis’s recent call for a FINRA-style pre-release standards body (linked below), alongside internal accounts of safety proposals stalling under commercial pressure at some of the labs writing these frameworks. But that’s a policy-layer conversation. The gap enterprise architects actually have to close is the one three levels below it: the specific sandbox, the specific credential, the specific network path that let an optimization process do in hours what a human red team would need weeks to find.

Thoughts and comments? Do share below.

References

Identity & Access Management for Agentic AI — Our Technical Report Is Now Published

Over the past three+ months, my co‑author Angelika Steinacker and I have been deep in the weeds researching, brainstorming, threat‑modeling, and refining what a secure identity and access architecture should look like in the era of agentic AI. Today, I’m excited to share that our technical paper Governing AI Agents – An Agent-Aware IAM Framework, is now publicly available.

👉 Read it on ResearchGate: https://www.researchgate.net/publication/400396082_Governing_AI_Agents_An_Agent-Aware_IAM_Framework

Why we wrote this

Agentic AI systems introduce Autonomous Non‑Human Identities (A‑NHIs)—entities that operate with autonomy, make decisions at machine speed, and collaborate across applications, APIs, and other agents. These behaviors fall far outside what traditional IAM was designed to handle.

Across our research, we observed consistent gaps in current IAM systems:

  • Reliance on static credentials
  • Lack of fine‑grained, purpose‑aligned authorization
  • Limited visibility into multi‑hop agent delegation chains
  • No robust way to establish dynamic cross‑domain trust
  • Insufficient mechanisms for end‑to‑end provenance

What this paper contributes

We propose an Agent‑Aware IAM model built on extending and fully implementing the Identity Fabric. The result is a four‑layer deployment architecture designed specifically for agentic environments:

  1. Identity Foundation — verifiable agent identities, ephemeral issuance, ownership, and purpose metadata
  2. Trust & Federation — dynamic cross‑domain trust using VCs, DIDs, token exchange, and trust brokers
  3. Security & Privacy Enforcement — intent‑aligned authorization, JIT access, privacy safeguards, and drift detection
  4. Lifecycle & Observability — full provenance: agent → token → task → data → decision

We illustrate these layers through a credit‑scoring + order‑management multi‑agent system, showing how secure, audited flows can be constructed end‑to‑end.

A collaboration worth highlighting

This work came from months of intense technical deep‑dives, design sessions, and constant iteration. Collaborating with my co‑author Angelika Steinacker made this intellectually exciting and extremely rewarding — discussions ranged from identity proofs and decentralized trust to model attestation, SBOM linkage, and federated governance.

Looking ahead

As enterprises move toward multi‑agent ecosystems, we believe trust—not raw capability—will define what can scale safely. Identity, policy, and provenance must become the control plane for autonomous digital workflows.

As I mentioned in my previous blog post Rethinking Identity in the Age of Multi-Agent Systems, this is a very important field of study, within the Agentic AI Systems realm. And there will be more work we need to do, as Security Architects, to ensure these Agentic systems operate within boundaries we set for them.

Thank you to everyone who encouraged this work along the way.
I hope this Paper serves as a useful reference for Enterprise Security Architects, CISOs, IAM teams, and AI governance practitioners navigating this emerging space.

Rethinking Identity in the Age of Multi-Agent Systems

Over the past few months, a recurring theme has emerged in my conversations with enterprise architects and CxOs across industries: “how do we prepare for the identity explosion that autonomous systems are bringing”.

As organizations begin deploying multi-agent systems (MAS) — collections of AI agents collaborating across environments — the familiar boundaries of Identity and Access Management (IAM) are being tested. Our IAM foundations were built around humans and static services. In contrast, non-human identities (NHIs) — the agents themselves — are transient, autonomous, and capable of making complex decisions without direct human oversight.

Many of my peers in the industry are already seeing the cracks. CxOs express growing concern about compliance and auditability: “Who authorized that action if no human clicked approve?” , “Who’s accountable when an agent takes an action no human explicitly approved”. Enterprise architects talk about the operational strain of managing thousands of short-lived agent credentials — each spun up dynamically, each needing verifiable provenance and revocation. Security leads worry about a new kind of “shadow identity” risk, where agents operate outside the current IAM visibility model.

Why Traditional IAM Architectures are not suitable for Agentic Systems

Identity Persistence vs. Agent Ephemerality:
Conventional IAM systems rely on static or semi-persistent identities (users, service accounts, API keys). Agentic systems operate with ephemeral, rapidly instantiated agents whose lifecycles may last seconds. IAM must evolve toward ephemeral credential issuance, context-bound authentication, and automated revocation tied to runtime telemetry and agent state.

Static Policy Models vs. Adaptive Agent Behavior:
Role- and attribute-based access control (RBAC/ABAC) frameworks assume stable roles and predictable intent. Agentic AI introduces goal drift and behavioral evolution, requiring adaptive authorization models driven by continuous policy evaluation, reinforcement signals, and runtime behavioral baselining.

Opaque Audit Trails vs. Cryptographically Verifiable Provenance:
Traditional logging mechanisms cannot reconstruct complex, multi-agent decision chains. Future IAM must embed verifiable provenance — linking every action to a unique agent identity, signed attestation, and timestamp — enabling non-repudiation, forensic replay, and accountability across distributed agent networks.

Static Privilege Boundaries vs. Autonomous Escalation:
Agents can probe environments and autonomously grant or delegate privileges via exposed APIs or inter-agent collaboration. This necessitates real-time privilege attestation, continuous risk scoring, and collusion detectionmechanisms to enforce least privilege dynamically.

Human-Centric Trust Models vs. Machine-Driven Collaboration:
Current IAM protocols (OAuth2, OIDC, SAML) were designed for human–service or service–service trust. In multi-agent ecosystems, we need machine-to-machine trust fabrics using Decentralized Identifiers (DIDs), Verifiable Credentials (VCs), mutual TLS, and zero-trust inter-agent authorization to maintain integrity across autonomous communication channels.

Figure: Key priorities for managing NHIs in a MAS

Recent Research

Recent research is formalizing the standards required for this shift, characterizing the current period as the Protocol-Oriented Interoperability phase (2024–2025). Addressing the delegation challenge, the IETF published a draft in May 2025 for an OAuth 2.0 Extension: On-Behalf-Of User Authorization for AI Agents. This extension introduces parameters like requested_actor and actor_token to authenticate the agent and document the explicit delegation chain in access tokens. Concurrently, protocols like Agent-to-Agent (A2A) for peer communication and the Model Context Protocol (MCP) for secure tool invocation are maturing. Furthermore, evaluating the ontological robustness of agents is being standardized through frameworks like Agent Identity Evals (AIE), which measure stability properties such as continuity, consistency, and recovery.

Looking ahead

I see this as a challenge but also a great opportunity, for us security architects: we need to reimagine identity from first principles — designing for autonomous, adaptive, non-human actors. This isn’t about extending old IAM models; it’s about building new trust fabrics grounded in cryptographic provenance, dynamic intent, and zero-trust collaboration. The architectures we design today will determine not only how securely these agents operate, but how trust itself is represented, delegated, and enforced in the digital ecosystems of the future.

As enterprises and societies and our civilization eventually grows increasingly dependent on intelligent systems, identity becomes the new fabric of trust. When machines act alongside us, the question isn’t just how we secure them — but how we preserve trust, accountability and intent in a world where human and machine agency converge….isn’t it?