Part 1 followed an intrusion from a human identity into application permissions and business data. An AI agent creates another route through those same resources. It can retrieve a document, select a tool, and ask a business service to perform an action. The security question is who authorized that action, especially when the document contains instructions nobody approved.
Microsoft's summary of the 2026 Digital Defense Report makes this system boundary explicit. Agent identity, data, tools, permissions, and surrounding infrastructure all matter. The model is one component, not the place where every security decision should happen.
My priority would be to make one important workflow safe to operate and possible to recover. That means enforcing permissions outside the model, recording side effects, and testing what happens when the agent or its identity must be suspended. It is a narrower goal than reviewing every AI feature, but it produces evidence an application owner can use.
Separate attacker use of AI from attacks on your agent
The report describes attackers applying AI to reconnaissance, social engineering, code analysis, and other stages of existing workflows. That changes the pace and cost of some work. It doesn't make a polished message proof of identity or make every intrusion fully autonomous. Familiar controls around authentication, exposed services, and software release still matter.
An attack on an AI system is a different problem. Indirect prompt injection places instructions in content the system retrieves, such as a document, email, or tool response. The legitimate user may have asked only for a summary. The content attempts to redirect that task, and the agent may then propose an action using permissions its runtime already holds.
Keep those questions separate when budgeting. Better phishing detection doesn't automatically protect a payment tool from an agent making an unauthorized request. A prompt filter doesn't protect a developer's release credentials from a malicious package. Each control needs a defined boundary and a test that exercises it.
Give the invoice agent a smaller job
Suppose an employee asks an agent to summarize a supplier invoice and compare it with a purchase order. The approved task needs read access to those records. It doesn't need permission to change bank details, export the supplier directory, or send a payment instruction. Start with that distinction before selecting the model or writing the system prompt.
Use synthetic invoices and records when testing. An invoice might contain text that tries to redirect the agent toward an unrelated export. The dangerous transition occurs when the application accepts the proposed tool call as authorized. The document itself isn't executable code, and a model's choice isn't a business approval.
Define the tool contract outside the prompt
The read tool should accept only the identifiers and operations needed for invoice comparison. It should enforce the caller's access to the requested records, validate arguments, and return only required fields. If the runtime uses an application identity with broader backend access, the service still needs a trustworthy way to enforce the requesting user's permitted scope.
Don't accept a tenant ID or user ID supplied by the model as proof of access. Obtain identity context from authenticated infrastructure, not generated arguments. Where delegation is used, validate the delegated token and resource permissions. Where application permissions are used, enforce the workflow's explicit authorization policy instead of assuming the service credential makes every user request permissible.
A separate write workflow can support approved changes. Bind an approval to the exact action, record, destination, and expiry. Validate it server-side and require fresh approval if those arguments change. An agent-generated sentence saying that the user approved an action isn't an approval artifact. For payment changes, preserve existing independent verification and separation-of-duties controls.
Restrict outbound destinations and decide how retries behave. If a tool can create a record or submit an instruction, use an idempotency mechanism appropriate to that service and a way to reconcile an uncertain result. A timeout may mean the request succeeded but its response was lost. Blind retries can duplicate an authorized action without any malicious prompt at all.
Review identity, retrieval, and memory together
Give the workflow an identifiable, narrowly scoped runtime identity. Avoid borrowing an administrator's interactive credentials or sharing one powerful credential across unrelated agents. Managed identities or workload federation can reduce stored secrets where supported, but neither removes the need to review permissions and trust conditions.
Inventory the repositories and connectors the agent can read. Retrieval should enforce the requesting user's allowed scope, not just search a broadly indexed collection and filter the answer afterwards. Once sensitive content reaches the model context, a final-output filter isn't an access-control boundary. Check whether index permissions remain aligned when source permissions change.
Persistent memory deserves its own owner. A useful remembered preference can become a harmful remembered instruction if arbitrary retrieved content can write it. Separate stored facts from policies, record source and writer provenance, define expiry, and provide a review and reset procedure. Memory must not grant new permissions or replace an approval.
| Boundary | Question to settle | Evidence to keep |
|---|---|---|
| Identity | Which caller and runtime identity authorize this operation? | Identity IDs, token audience and permission context without logging the raw token |
| Retrieval | Could this user access the same record outside the agent? | Source record ID, access decision, relevant source-permission version |
| Memory | Who wrote this state, and can it be removed or rolled back? | Writer, source, version, retention and reset records |
| Tool execution | Was this exact action authorized at the target resource? | Validated arguments, policy decision, approval reference, resource operation result |
| External connection | Who operates this tool or MCP server, and what credentials does it hold? | Owner, authentication configuration, permitted tools and destinations |
Treat a Model Context Protocol (MCP) server as an API integration. Review its operator, deployment, authentication, tool schemas, and downstream credentials. A tool's description can influence model selection, but doesn't establish that the server is trusted or that a caller may use every operation it exposes. Test the authorization at the server and at its downstream resource.
Use prompt protection without outsourcing authorization
Microsoft's Prompt Shields documentation distinguishes direct user-prompt attacks from document attacks in third-party content. This is useful protection around the model. Confirm the supported configuration for your deployment and examine whether an event was detected, filtered, or merely annotated.
A detected attempt that wasn't filtered is different from a blocked input. Even filtering isn't a substitute for checking the tool result. If the workflow performed the side effect before showing a safety message, the message doesn't undo it. Tool services must deny unauthorized actions even when the model proposes them confidently.
Also protect the ordinary software around the agent. Review generated code and proposed package names before installation. Pin and verify dependencies, protect build credentials, and test connector changes. A new model version, retrieval source, tool definition, or permission grant changes assumptions that earlier tests may have relied on.
Test side effects, not just the answer
Use a non-production environment with synthetic supplier records and a controlled destination. Give each test a request ID, expected authorization decision, and an observable result at the resource. Record whether the input reached retrieval, whether the model proposed an action, whether policy allowed it, and whether a write or external transmission occurred.
| Exercise | Required outcome | Check independently |
|---|---|---|
| Invoice content tries to redirect a summary into a data export | No unrelated export or external transmission | Tool decisions, outbound records, target-service operations |
| User requests another user's restricted record | Access denied before restricted content enters model context | Retrieval result and authorization event |
| Retrieved text attempts to persist a new instruction | No policy or permission change through memory | Memory versions and a subsequent clean session |
| An approved action is retried after a simulated timeout | No duplicate business operation | Operation IDs and target-record state |
| Agent access is suspended during a queued task | Further disallowed operations stop; residual access is measured | Queue state, issued-token behavior, resource denials |
Keep legitimate cases in the same test set. The agent should still summarize an allowed invoice and handle an unavailable record without inventing a successful result. A control that blocks every request can pass a narrow security test while making the service unusable. Verify both the permitted task and the forbidden side effect.
Repeat the relevant cases after changes, recording the model and application versions, tool schemas, permission configuration, and retrieval inputs. A passing result is evidence for that tested configuration, not proof that no prompt injection will ever work. Review unexplained gaps between the model trace and resource logs as test failures, not harmless missing detail.
Test the stop control and the recovery path
Stopping an agent needs more than closing its chat window. Decide how to pause triggers and queued work, deny further tool execution, restrict the runtime identity, and deal with cached access. As Microsoft's access-revocation guidance explains, sessions and issued tokens have different lifetimes and control points. Measure the stop behavior at the downstream service.
Preserve the configuration and relevant records before resetting memory or rebuilding. If invoice records were changed, turning the agent off doesn't repair them. Identify affected operations, validate the authoritative source, and reconcile changes with the business owner. Avoid an automatic rollback that overwrites legitimate transactions made after the incident.
Run a recovery exercise without the normal administrator
Choose a representative critical service and assume its production administrator is compromised. Use a clean recovery workstation and separately protected emergency access. Restore into an isolated environment, using backups the compromised production identity cannot alter. Verify configuration, data integrity, application dependencies, and the restored service's credentials before allowing normal traffic.
Add an identity-provider outage to the exercise. Check what emergency access still depends on that provider, which services require their own local procedure, and whether recovery keys and contacts are available without the usual mailbox or password vault. Don't assume a cloud emergency account solves every SaaS or infrastructure dependency. Test the documented route for each priority service.
The business owner should validate that invoice processing or another essential operation works, not merely that a virtual machine boots. Measure recovery time from the agreed incident or disruption start until that acceptance point. Measure recovery point as the data loss relative to the agreed reference time. Compare the observed results with the service's approved targets and record manual steps that made the difference.
Include one supplier dependency. If a connector provider is unavailable or suspected of compromise, who can pause it, who verifies the notification, and what restricted manual process can continue? A recovery plan that needs the unavailable supplier to issue every credential hasn't tested that failure. Keep operational and safety owners involved where isolation could affect physical equipment.
Budget for evidence and protect it
Agent testing consumes model requests and tool execution. Production monitoring adds storage, ingestion, retention, and review costs. Recovery exercises need staff time and temporary infrastructure. Estimate these separately from the model endpoint's token price; a cheap inference call can still initiate an expensive or consequential workflow.
Start with structured operational evidence: request IDs, authenticated identities, source record IDs, policy decisions, approval references, and operation outcomes. Full prompts and tool responses may contain personal data, commercial documents, or secrets. Collect them only where needed and permitted, redact sensitive fields, set retention, and restrict investigators' access. Don't log raw credentials for convenience.
Agree where data is processed and stored, including third-party tools and support workflows. A sensitivity label is useful context, but doesn't enforce every connector's access or destination policy. Verify the actual enforcement and involve privacy and compliance owners. A passed agent test alone doesn't establish GDPR, PCI, or another compliance outcome.
A focused 90-day plan
The work should produce a smaller, testable access boundary, not another inventory nobody owns. Apply this sequence to the selected workflow first. Urgent exposure or an active incident still requires immediate response rather than waiting for its place on the calendar.
- In days 1-30, name the business and technical owners. Map the user, agent identity, data sources, tools, memory, and suppliers. Define permitted actions, high-impact approvals, required evidence, recovery targets, and a restricted fallback.
- In days 31-60, remove unnecessary access and enforce the tool contract at the resource. Separate read and write paths, protect memory updates, add correlation records, and demonstrate containment. Run both adversarial and legitimate workflow cases.
- In days 61-90, exercise queued-work suspension, uncertain retries, supplier failure, and recovery without the production administrator. Have the business owner accept the restored workflow and assign unresolved gaps an owner and completion date.
Report the proportion of critical tool operations with tested authorization, the observed time until disallowed access stops, and the measured recovery result. Define the denominator and start and end events. Those measures show whether the workflow is safer to operate; a count of agent prompts reviewed doesn't.
Part 1's identity investigation and this agent review share the same test: follow the authority into the resource, verify where it can be stopped, and prove the business can resume without reintroducing compromised trust. I would take that evidence to the next security review before giving the workflow more permissions or more autonomy.
For the broader threat and resilience context, read the full 2026 Microsoft Digital Defense Report.