Module 12

Agentic AI and AI Systems

Design, evaluate and govern AI agents that use tools and take actions — with the controls that make them safe enough to ship.

≈ 12 h studyDraftedSources: Lectures 46, 48, 49, 50, 51, 54 (unavailable), 55; Picture lessons (agents, API/MCP); Companion ch. 14, 15, 25, 27

Mark module complete Expand all

On this page

1. What an AI agent is
2. Agents versus chatbots versus workflows
3. Autonomous versus human-supervised systems
4. Agent architecture
5. Planning and reasoning workflows
6. Tool calling and function calling
7. Memory systems
8. Multi-agent systems and orchestration
9. Workflow automation
10. RAG-enabled agents
11. Evaluation and observability for agents
12. Human approval checkpoints, guardrails and permission boundaries
13. Agent security and prompt injection
14. Latency and operating costs
15. Failure recovery
16. Agentic product design
17. AI automation opportunities in finance
18. AI product roadmaps and agentic business models
19. Case study: AI-powered customer-support assistant for a card issuer (hypothetical)
20. Exercises with model answers
21. Interview perspective
22. Key takeaways
23. Check your understanding
24. Recommended resources

12.1 What an AI agent is

An AI agent is a system that uses a model to pursue a goal by choosing actions — calling tools, reading results, and deciding what to do next — within limits set by people or policy.

12.2 Agents versus chatbots versus workflows

Common composable patterns (from Anthropic's “Building effective agents”): prompt chaining (step-by-step pipeline), routing (classify then send to a specialised path), parallelisation (split or vote), orchestrator–workers (a lead model delegates subtasks), evaluator–optimiser (one generates, another critiques), and autonomous agents (open-ended loops with tools).

12.3 Autonomous versus human-supervised systems

12.4 Agent architecture

12.5 Planning and reasoning workflows

• ReAct (reason + act): the model alternates thinking about the next step and calling a tool, using observations to update its plan.
• Plan-then-execute: draft a plan, check it (by rules, another model or a human), then execute steps.
• Reflection / self-critique: evaluate intermediate outputs before continuing.
• Decomposition: break a large goal into sub-goals with clear completion criteria.

PM implication: plans must be observable (logged) and checkable — reviewers should be able to see why the agent did something.

12.6 Tool calling and function calling

The model is given tool definitions (name, description, input schema). When useful, it outputs a structured call; the application executes it (with permission checks) and returns the result for the model to use.

{
  "name": "get_card_status",
  "description": "Return the status of the authenticated customer's card. Read-only.",
  "input_schema": {
    "type": "object",
    "properties": { "card_last4": { "type": "string", "pattern": "^[0-9]{4}$" } },
    "required": ["card_last4"]
  }
}

Design rules: small, single-purpose tools; clear descriptions (the model reads them); separate read and write tools; validate inputs server-side; return structured errors; make write tools idempotent; require approval for consequential writes.

12.6.1 MCP and APIs

12.7 Memory systems

Product rules: tell users what is remembered; let them view/delete; never store sensitive data without purpose and consent; expire memories; keep authoritative data in systems of record, not in the agent's memory.

12.8 Multi-agent systems and orchestration

Multi-agent systems split work among specialised agents (e.g., a triage agent, a policy-lookup agent, a drafting agent, a reviewer agent) coordinated by an orchestrator. Benefits: specialisation, parallelism, separation of permissions. Costs: more tokens, latency, harder debugging, error propagation between agents.

12.9 Workflow automation

Tools like n8n, Zapier or Make connect triggers and actions — often with AI steps inside a deterministic flow. This is frequently the right answer before a fully autonomous agent.

12.10 RAG-enabled agents

An agent can call retrieval as a tool: decide whether to search, what to search, and whether results suffice. Benefits: answers grounded in policy; can combine knowledge with account data. Risks: retrieved text may contain indirect prompt injection; permissions must filter what is retrievable for this user (Module 10).

12.11 Evaluation and observability for agents

Evaluate at three levels:

1. Outcome: did the task get done correctly? (task success rate, correct resolution)
2. Trajectory: were the steps sensible? (right tools, right order, no unnecessary or unsafe calls, number of steps)
3. Component: retrieval quality, tool-call argument accuracy, refusal/escalation correctness.

Observability: trace every run (prompts, tool calls, arguments, results, latency, tokens, costs, decisions); dashboards for success, escalation, errors, cost per task; sample reviews; alerting on anomalies (spikes in tool errors or refusals).

12.11.1 Reliability compounds

12.12 Human approval checkpoints, guardrails and permission boundaries

• Approval checkpoints: before money moves, account details change, messages go to customers, or irreversible actions happen. Show the reviewer what will happen, why, and the evidence.
• Guardrails: input filters (PII, abuse, injection patterns), output filters (policy compliance, PII leakage), schema validation, allow-lists of actions, spending/time/step limits, rate limits, kill switch.
• Permission boundaries: least privilege per tool; scoped tokens; per-user authorisation enforced by the backend; separate read and write credentials; sandbox and test environments for development.

12.13 Agent security and prompt injection

Prompt injection is when untrusted text (a customer message, a web page, an email, a document) contains instructions that hijack the agent. Indirect injection hides them in content the agent retrieves.

The most dangerous combination — described by security researcher Simon Willison as the “lethal trifecta” — is an agent with (1) access to private data, (2) exposure to untrusted content, and (3) the ability to communicate externally. Remove at least one leg for sensitive workflows.

Limit capability

• Least-privilege tools
• No external send without approval
• Separate agents for untrusted content

Isolate untrusted text

• Delimit and label as data
• Never treat retrieved text as instructions
• Strip hidden/markup instructions

Validate actions

• Allow-lists and schemas
• Policy checks before execution
• Human approval for consequential actions

Detect & monitor

• Injection classifiers (imperfect)
• Anomaly alerts on tool use
• Red-team regularly

Contain damage

• Scoped, short-lived tokens
• Idempotent, reversible actions
• Audit logs

Test

• Adversarial eval cases in every release
• Include indirect injection via documents and emails

12.14 Latency and operating costs

12.15 Failure recovery

• Retries with back-off for transient errors; idempotency keys so retries don't duplicate actions.
• Checkpoints and state so a long task resumes rather than restarts.
• Timeouts and step budgets to stop runaway loops.
• Compensating actions / rollback for multi-step changes (e.g., reverse a hold if a later step fails).
• Graceful handoff to a human with full context when the agent is stuck.
• Post-incident review feeding new eval cases.

12.16 Agentic product design

Design for trust and control:

• Transparency: show what the agent is doing (“Checking your last 3 transactions…”), what it used (sources), and what it will do next.
• Control: preview and approve; easy cancel; undo where possible.
• Scoping: tell users what it can and cannot do.
• Progressive autonomy: start with suggestions, earn the right to act.
• Feedback loops: thumbs, corrections, and escalation feed evaluation.
• Accessibility and inclusion: multiple languages, clear wording, alternative channels.

12.17 AI automation opportunities in finance

12.18 AI product roadmaps and agentic business models

Roadmap shape: (1) internal copilot for staff (L0–L1) → (2) customer self-service for low-risk intents with RAG (L2) → (3) transactional actions within limits with approvals (L3) → (4) proactive agents (alerts, nudges) once trust and monitoring are proven. Each stage has evaluation gates.

Business models: per-seat (copilots), usage-based (per task/token), outcome-based (per resolved conversation or per successful action), tiered bundles. Outcome pricing aligns value but requires agreed definitions of “resolved” and robust measurement. Internally, ROI = (cost per human-handled contact − cost per AI-resolved contact) × resolved volume − platform and oversight costs — including the cost of errors.

---

12.19 Case study: AI-powered customer-support assistant for a card issuer (hypothetical)

12.19.1 Business background

A mid-sized (fictional) card issuer serves ~2 million cardholders through app chat, phone and email. Contact volume rises during sale seasons; customers wait long for simple answers (card block, statements, fees), while complex cases (disputes, fraud, hardship) need skilled agents.

12.19.2 Target users

• Cardholders seeking fast answers and simple actions.
• Support agents handling complex and sensitive cases.
• Operations/compliance teams who own policy and audit.

12.19.3 Customer problem

Customers wait for answers to simple questions and must repeat information when transferred. Agents spend time on routine queries and copying data between systems.

12.19.4 Business objective

Reduce cost per contact and wait times without increasing complaints, errors or compliance breaches.

12.19.5 Available evidence (hypothetical)

• Contact reasons for the last quarter (ticket tags): card block/unblock 14%, statement and fees 21%, payment status 12%, rewards 8%, disputes 9%, fraud 6%, hardship 3%, other 27%.
• Median chat wait 11 minutes at peak; 34% of chats transferred at least once.
• Policy documents exist but are inconsistent in places.

12.19.6 Assumptions to test

1. ≥ 40% of contacts are low-risk informational intents answerable from policy + read-only account data.
2. Customers will accept an AI answer if it is fast, cited and offers a human.
3. The assistant can recognise high-risk intents (disputes, fraud, hardship) reliably enough to route them.

12.19.7 Product hypothesis

We believe an assistant that answers low-risk intents from approved policies and read-only account data, and routes high-risk intents to agents with a summary, will cut median wait time by 50% and resolve 35% of chats without an agent, while keeping CSAT and complaint rates at or better than baseline.

12.19.8 User journey (target state)

12.19.9 Candidate solutions

12.19.10 Prioritisation

Using a weighted score (value 35%, risk 25% inverse, feasibility 20%, time-to-value 20%), the team picks D + B first: the copilot builds evaluation data and agent trust; the read-only assistant addresses wait times. C follows with one low-risk action (temporary card block, reversible) behind explicit confirmation.

12.19.11 MVP

• Channels: in-app chat for authenticated customers in one language.
• Intents: card status/block (read-only status), statement location, fee explanations, rewards balance — answer; disputes, fraud, hardship, complaints — route with summary; off-topic financial advice — refuse politely.
• Grounding: approved policy pages via RAG with citations; account facts via read-only tools.
• Controls: permission-aware retrieval; no external sends; PII redaction in logs; human handoff button always visible; kill switch.
• Pilot: 5% of eligible chats for 4 weeks; agents review a daily sample.

12.19.12 KPIs

• Primary: containment / self-service resolution rate (resolved without agent, no repeat contact in 7 days).
• Customer: median time to answer; CSAT for bot-handled chats; complaint rate.
• Quality & safety: policy accuracy on sampled answers; correct routing of high-risk intents (target ≥ 98%); PII-leak incidents (target 0); injection success on red-team set (target 0).
• Operational: agent handle time for routed chats (summary benefit); cost per resolved contact.

12.19.13 Risks and constraints

Regulatory obligations (complaints handling, disputes timelines, fair treatment, data protection); hallucinated policy; mis-routing vulnerable customers; prompt injection via customer messages; authentication and authorisation bugs; model/vendor changes; inconsistent policy sources; language and accessibility gaps.

12.19.14 Experiment plan

1. Offline eval on 60+ curated cases (see datasets/support_eval_set.csv) plus 500 anonymised historical chats labelled by agents; launch gate: routing accuracy ≥ 98% on high-risk intents and zero severity-1 failures.
2. Agent copilot pilot (2 weeks) to compare handle time and agent satisfaction.
3. Customer pilot: randomised 5% of eligible chats vs control (current flow), 4 weeks; guardrails reviewed daily.
4. Decision review with compliance and support leadership.

12.19.15 Example metrics (hypothetical pilot readout)

12.19.16 Product recommendation

Do not scale yet. Fix routing misses (most involved hardship language such as “I can't pay”), add those phrasings to the eval set, investigate the complaint increase, then extend to 20% of traffic. Begin design of the first action (temporary card block) with explicit confirmation and a reversal path.

12.19.17 Potential outcomes

• Best case: 35–40% resolution, wait times collapse for simple queries, agents focus on complex cases.
• Base case: 25–30% resolution with strong agent-copilot gains.
• Worst case: policy errors or mis-routing cause complaints/regulatory exposure → kill switch, revert to copilot-only.

12.19.18 Key PM lessons

1. Start with assistive, read-only capabilities and earn autonomy.
2. Routing high-risk intents is a launch-blocking capability, not a nice-to-have.
3. Evaluation sets and severity weighting drive decisions — not demos.
4. Permissions live in the backend, never in the prompt.
5. Measure customer outcomes and safety, not just containment.

---

12.20 Exercises with model answers

Model answer

A workflow with one AI step (field extraction): the path is known and auditable; matching and posting are deterministic rules. Add maker–checker review for exceptions and an approval for postings above a threshold. An agent adds unpredictability without benefit here.

Model answer (example)

Auto-approve refunds ≤ ₹500 when policy criteria are met and the customer has no recent refunds; require agent approval for ₹500–₹5,000 or any repeat refund in 30 days; require supervisor approval above ₹5,000. All refunds idempotent (refund ID), logged with evidence, reversible within policy, and sampled weekly for audit. Kill switch if refund rate exceeds a daily threshold.

Model answer

It has all three legs: untrusted content, private data and external communication — an injected email could make it exfiltrate customer data. Redesign: remove autonomous sending (drafts only, human sends), or split into a reader agent with no data access and a separate drafting step with read access but no sending; restrict searchable fields; add injection tests.

12.21 Interview perspective

“Design an AI agent for customer support at a bank.” — structure

Clarify goals and constraints (regulation, channels, languages) → segment intents by risk → choose autonomy per intent (answer, route, act with approval) → architecture (RAG + read-only tools + approvals, permission-aware) → evaluation plan (offline set, severity, red-team, pilot A/B) → metrics (resolution, CSAT, routing accuracy, safety incidents, cost) → rollout and governance (kill switch, audits). Mistakes: full autonomy on day one; ignoring vulnerable-customer routing; trusting the model with access control.

“How do you evaluate an agent?” — 60-second answer

“At three levels. Outcome — did it complete the task correctly, measured on a curated test set and in a controlled pilot. Trajectory — did it use the right tools in a sensible order without unsafe or wasted steps, which I check from traces. Components — retrieval quality, tool-argument accuracy, routing and refusals. I weight failures by severity, red-team for prompt injection, and re-run the suite on every model, prompt or tool change. In production I trace every run and sample reviews weekly.”

12.22 Key takeaways

• Prefer workflows to agents unless the path genuinely varies; add autonomy gradually with evidence.
• Agents = model + tools + memory + control loop + guardrails; enforce permissions in the backend.
• Reliability compounds across steps; reduce steps, add verification and recovery.
• Defend against prompt injection in layers; avoid the private-data + untrusted-content + external-action combination.
• Evaluate outcomes, trajectories and components; trace everything.
• Agent costs and latency are multiples of single calls — budget steps.

12.23 Check your understanding

12.24 Recommended resources

• Anthropic: Building effective agents (from your course references).
• Model Context Protocol: Introduction (course reference).
• OWASP Top 10 for LLM Applications.
• Simon Willison: The lethal trifecta for AI agents.
• n8n: Build your first workflow (course reference).
• NIST AI Risk Management Framework.