Category: Technical

There’s a particular kind of person who, when handed a shiny new AI course, immediately asks: “Yes, but can I reproduce it, and what happens when it fails at 2 a.m.?” That person does not get invited to many parties. That person is me, and this is my review.

DeepLearning.AI’s Agentic AI course walks you through five modules that build, brick by brick, from a single reflecting agent to a coordinated team of them: reflection → tool use → evaluation → multi-agent collaboration, all hung on one running example — a research-report agent — plus a rotating cast of standalone labs (chart bots, SQL bots, an email assistant, a sunglasses-store customer-service pipeline, a marketing team). Andrew Ng narrates. The slides are clean. The vibe is optimistic.

My job is to be the opposite of optimistic. So I studied every lecture PDF, ran a magnifying glass over every ungraded and graded lab, cracked open the autograders, and benchmarked the whole thing against what the rest of the field — Hugging Face’s Agents Course, LangGraph, CrewAI, AutoGen, Anthropic’s tool-use docs, LangSmith, DeepEval — teaches for the same money (which, for a lot of them, is zero).

The headline: as a set of ideas, this course is genuinely well-sequenced and honest. As a set of artifacts you might mistake for production code, it is a target-rich environment. The gap between those two sentences is the entire review.

Most intro modules hand you a notebook and a warm feeling. Module 1 hands you a deployable FastAPI + Postgres app in a Docker container and says good luck. A planner (o4-mini, temperature=1) emits a 5–7 step plan; an executor routes each step to a research/writer/editor agent; the research agents call Tavily, arXiv, and Wikipedia; the result is persisted to Postgres and streamed to a UI. Shipping a real back end on day one is genuinely unusual — Hugging Face opens with one smolagents notebook; Microsoft’s “AI Agents for Beginners” is concept-first lessons. This throws you in the deep end of deployment, state, and orchestration immediately.

Which is exactly why the validation findings pile up so fast.

The signature crime is on main.py:49: an unconditional Base.metadata.drop_all() on every startup. Every restart nukes the tasks table — the audit trail, the one thing a validator actually lives for, has the persistence half-life of a container. The README documents this as a “gotcha” rather than fixing it, which is a very honest way of saying “we know.” Around it: dependencies that float free (2 of ~30 packages pinned, so two builds a week apart are two different models of reality); a research_agent that swallows exceptions and returns "[Model Error: ...]" as content the writer will then cheerfully cite; routing by substring match (if "research" in step_lower), so a step titled “Write up the research draft” routes to the research agent because “research” got checked first; allow_origins=["*"]; default Postgres creds on an exposed port; and LLM-authored raw HTML rendered into innerHTML with no sanitization — stored XSS waiting for the wrong search result.

What’s genuinely instructive is the shape: the clean planner/executor/agents/tools separation, a real multi-source tool loop, and a Docker one-shot that actually runs end to end. What’s dangerous is that it looks production-ready enough for someone to treat it that way.

My priors on “the model critiques itself” are openly hostile, so I came in swinging. Module 2 mostly earned my grudging respect, because it stages the failure instead of hiding it.

The pattern is simple: emit V1, critique it, emit V2. The deck escalates through three grounding regimes — ungrounded self-reflection, execution feedback (run the code, feed back the stderr), and multimodal feedback (render the chart, let a vision model judge the actual pixels). Pleasingly, it doesn’t stop at “reflection good”: it cites Self-Refine (Madaan et al., 2023), then spends five slides on evaluation and explicitly warns about position bias in LLM-as-judge. Acknowledging that your evaluator is itself a biased instrument, in a beginner course, is more intellectual honesty than I budgeted for.

The star is the SQL lab (M2_UGL_2). It runs two reflection passes on a query over an event-sourced schema where qty_delta < 0 means a sale. The text-only refiner inspects the SQL, pronounces it fine, and confidently returns a negative total-sales number. Then the grounded refiner gets fed the actual result DataFrame, sees the nonsense, and fixes it with ABS(). That’s the whole model-validation lesson in one lab: self-reflection without an external oracle is theater.

Where it thins out is everything a validator asks next. No lab ever checks that V2 beat V1 — improvement is assumed, never measured. There’s no loop, no stopping criterion, no regression guard. The critic in the grounded SQL refiner runs at temperature=1.0 (a nondeterministic validator — a hard sell in any governed pipeline). And the graded assignment is the least trustworthy instance of the whole pattern: ungrounded essay critique, autograded only on “returns a str, no exception, length > 100.” You can pass with three near-identical three-line f-strings and never once observe whether reflection did anything. Defensible — free text can’t be autograded deterministically — but a cop-out all the same.

Versus the field: DLAI is more practical on the grounding distinction than most, but shallower on iteration — it never mentions Reflexion’s cross-trial memory, and LangGraph frames reflection as a stateful graph with loops and stopping edges. DLAI stops at V2; real systems iterate to convergence.

This is the module I actually lose sleep over, because tool use is the exact moment a text predictor acquires side effects on the world. The mechanics are taught cleanly — a tool is just code the LLM can request; here’s naive string-parsing, here’s the JSON function-calling contract, here’s multi-tool orchestration, here’s MCP as the m×n → m+n integration story. Two calling conventions (raw OpenAI vs. aisuite), which is realistic but plants the provider:model footgun ("gpt-4o" vs "openai:gpt-4o") that the README rightly flags as the #1 cause of silent NoneType has no attribute 'choices' failures.

And then the email lab (M3_UGL_2) hands the model a delete_email tool. I read the backend: email_service.py:194 is db.delete(email); db.commit() — a hard delete. No soft-delete, no trash, no tombstone; recovery means reseeding the fixture. Worse, the notebook’s own prompt instructs: “Never ask the user for confirmation before performing an action… (no human-in-the-loop).” The lab hard-codes the anti-pattern — full autonomy over a destructive verb — and then the closing cell congratulates you on building agents that act “safely and transparently.” It demonstrates transparency. It does not demonstrate safety, and conflating the two is precisely the trap a risk function exists to catch.

Elsewhere: web and arXiv content flows straight into the message history — a textbook indirect prompt-injection surface, never mentioned. Argument validation is fully delegated to the model (delete_email(email_id) trusts whatever integer the LLM hallucinates; no existence or ownership check). The autograder, despite the README claiming it “mocks the client,” actually calls your function live and asserts isinstance(out, str) and len(out) > 50 — 51 characters of lorem ipsum passes.

Credit where it’s due, and it’s real credit: auditability is the module’s genuine strength. The assignment forces you to hand-build the tool-response messages and retain the full history, and pretty_print_chat_completion renders the entire tool-call trace. You can reconstruct which tool fired with what arguments — the closest thing to a model-risk audit log in the whole course.

Versus the field, this is where DLAI trails hardest. Hugging Face names web-content injection as a memory-poisoning attack; Anthropic’s docs say “require confirmation for destructive operations”; LangGraph makes interrupt-on-destructive-action a first-class primitive. Module 3 teaches the literal opposite default and never shows the alternative exists.

This is home turf, so naturally it’s the module I graded hardest. And here’s the twist: the framing is the best thing in the entire course. The 2×2 of objective-vs-judge crossed with ground-truth-vs-none is genuinely reusable pedagogy. Error analysis via traces — count which stage produces the bad output (search results 45%, source selection 10%…) and spend your budget there — is just attribution analysis, and it’s the right instinct. The evaluation-vs-reflection distinction the README draws (reflection fixes this run; evaluation is a reusable scorer that catches regressions across many runs) is exactly the offline-eval-as-regression-gate idea a validator wants to hear.

Then the labs show up and start filing exceptions against themselves.

LLM-as-judge is introduced and simply trusted — no calibration against human labels, no inter-rater agreement, no acknowledgement that a judge from the same model family as the generator is marking its own homework, no treatment of the scorer’s own non-determinism (same rubric, same input, different score — sampling noise cosplaying as a P&L improvement). “Trustworthy source” is operationalized as “URL host is on an allowlist,” which quietly swaps provenance for correctness — a well-sourced wrong answer scores a clean PASS, and arxiv.org is un-reviewed preprints. And the matching in the notebook students actually run is substring, not suffix (any(td in domain for td in TOP_DOMAINS)), so nature.com would happily trust nature.com.phishing.io. The utils.py next door does it correctly with endswith. The one they run does not.

The assignment is where it falls over completely. There’s no blank template to complete — only a filled-in submission.ipynb that’s almost entirely mocked, with TOP_DOMAINS = {} (an empty dict, so validation is structurally guaranteed to FAIL). And the connoisseur’s detail, the thing I could not have designed as satire: grader.py’s part_1 builds cases = [] and returns it. The autograder for the evaluation module evaluates nothing. The validator’s validator is a stub.

What a real validation function would demand and this omits entirely: a versioned held-out eval set, ground truth beyond a domain list, confidence intervals (n≈20 with no error bars is a vibe, not a measurement), human-calibration of the judge, drift monitoring, and trajectory-level metrics (τ-bench’s pass^k reliability, tool-call correctness). The field — LangSmith, DeepEval’s 14 metrics, Ragas, Phoenix — treats agent eval as the hard problem it is. Even DLAI’s own “Evaluating and Debugging Generative AI” goes deeper.

The capstone hands you an org chart: a team of specialized agents — planner, researcher, writer, editor — passing work down a line, coordinated by an executor that reads each plan step and routes it to whoever should do it. The slides are the most intellectually interesting of the course: linear pipelines, then all-to-all, then deeper hierarchies (manager → sub-teams → fact-checker → citation-checker). It’s a lovely tour. The labs only ever build the linear case. You’re shown the hard topologies and handed the easy one.

Here’s the number that belongs on slide one and isn’t: if each agent in a four-step chain is 90% reliable, the chain is 0.9⁴ ≈ 66%. The entire selling point of multi-agent — decomposition — is also its central risk, because errors propagate and compound rather than cancel. The course never quantifies this, so a learner leaves thinking “more agents = more capability” when the validation reality is “more agents = more independent failure points in series.”

And the routing layer, the load-bearing wall of the whole design, is made of drywall. The executor_agent asks gpt-4o for “only a valid JSON object,” runs the reply through clean_json_block() — a regex that scrapes off Markdown code fences — and then calls a bare json.loads() with no try/except. The very existence of that fence-scraper is a confession that the model doesn’t reliably return clean JSON, so we bolt one on and pray. The day it emits a stray sentence or a trailing comma, the entire pipeline throws and dies mid-report. “Our orchestration layer deserializes free-text LLM output with no schema validation and no fallback” is not a design; it’s an incident with a pending date.

But — and this is real — M5_UGL_1 is the best-engineered lab in the entire course. A sunglasses-store customer-service pipeline with actual inventory/transactions tables in pandas + DuckDB, real tool functions, a planning step (request → JSON tool plan), a reflection step that repairs a flawed plan against a TOOLS-ONLY spec, and an error-explanation step that turns “this return would drive stock negative” into human guidance. It’s the closest the course comes to production shape. (It still explains the failure rather than preventing it with a rollback — but explaining it is more than most demos manage.) The campaign-team lab (M5_UGL_2) is the fun one — researcher → designer → copywriter → packager compiling an HTML report — but it’s a linear happy-path where every agent trusts its upstream blindly, with no evaluation gate between stages.

Versus the field, the comparison is almost unfair, because DeepLearning.AI has dedicated multi-agent courses — “Multi AI Agent Systems with crewAI,” “AI Agentic Design Patterns with AutoGen.” Against those, plus LangGraph’s stateful graphs with checkpoints and retries and OpenAI’s Agents SDK handoffs, this single capstone frames orchestration as a prompt-engineering problem solved with a hand-rolled JSON router, where the serious tools frame it as a reliability problem with real primitives. As a first exposure to the idea, it’s clear. As preparation to ship one, it’s a trailhead, not a map.

In which I build the perfect AI-course reading list, then do to it what I’m paid to do to everyone else’s models.

The confession

There is a particular irony in a model-risk quant deciding to learn how to build AI agents. My job is to sit across the table from someone who has built something clever and ask, as politely as I can manage, “Yes, but how does it break?” I am paid to be the person who doesn’t get swept up in the excitement. Skepticism isn’t a mood for me; it’s a line item.

So when large language models went from party trick to production dependency in roughly the time it takes to run a Monte Carlo simulation, my first instinct was the usual one: fold my arms and wait for the backtest. But a funny thing happens when you spend enough years reviewing other people’s models. You realize the only thing worse than a technology you distrust is one you distrust and don’t understand. You cannot challenge what you cannot read.

So I did the thing skeptics rarely admit to doing. I decided to actually learn it, hands on keyboard, from people who build these systems for a living rather than from a thousand breathless LinkedIn posts. This is the story of the syllabus I built to do that — and, more usefully, the story of how I turned around and validated my own reading list before trusting it.

What’s actually on DeepLearning.AI

If you haven’t looked lately, DeepLearning.AI has quietly become the place practitioners go to learn this stack. It’s Andrew Ng’s shop, and it shows: the pedagogy is unfussy, the notebooks run, and the instructors are people who ship.

The catalog splits into two shapes. There are the foundational specializations, multi-week affairs that teach a subject properly, and then there are the short courses, of which there are now around 124. The short courses are the real draw. Most run one to three hours, each is taught by a named practitioner from a company you’ve heard of, and every one hands you a code notebook so you’re typing rather than nodding. Anthropic, Google, Meta, LangChain, crewAI, LlamaIndex, Hugging Face, Nvidia, Databricks: the guest list reads like the org chart of the entire field.

The topics span the whole territory: prompting, the internals of transformers, retrieval-augmented generation, single and multi-agent systems, evaluation, guardrails and red-teaming, fine-tuning, and coding agents. It’s subscription-based, which for someone who binges in bursts felt fair. The catch is that with 124 courses and no obvious ordering, it’s easy to wander in and drown.

Why I built a five-tier path

Drowning is the problem a curriculum solves. Left to its own devices, a catalog this large becomes a buffet where you fill your plate with whatever’s shiny and leave still hungry for the basics. I wanted a spine, an order that built competence deliberately instead of accreting trivia.

Five tiers, roughly: foundations first, then the practical craft of building agents, then, crucially, evaluation, governance, and reliability, then the supporting technical skills that make the rest robust, and finally some optional productivity tooling.

Here’s where being a risk person rearranges the furniture. In most self-taught AI curricula, evaluation and guardrails are the vegetables you’re meant to eat after the fun part. For someone in model risk, that ordering is backwards to the point of being funny. Evaluation, governance, red-teaming: that isn’t the compliance tax on the interesting work. That is the interesting work. An agent that takes autonomous, multi-step actions is, from where I sit, a model with a much larger blast radius and far worse documentation. So I gave that material its own tier and refused to treat it as an afterthought.

Draft one: the list that looked perfect

Here is what I landed on first. Read it and tell me it doesn’t look like a serious, well-rounded plan:

• Tier 1 — Foundations. Agentic AI (DeepLearning.AI, Andrew Ng); How Transformer LLMs Work (Jay Alammar & Maarten Grootendorst).
• Tier 2 — Core agent-building. MCP: Build Rich-Context AI Apps with Anthropic (Anthropic); AI Agents in LangGraph (LangChain/Tavily); Building Agentic RAG with LlamaIndex (LlamaIndex); Multi AI Agent Systems with crewAI (crewAI); A2A: The Agent2Agent Protocol (Google Cloud/IBM Research).
• Tier 3 — Evaluation, reliability, governance. Evaluating AI Agents (Arize); Governing AI Agents (Databricks); Safe and Reliable AI via Guardrails (GuardrailsAI); Red Teaming LLM Applications (Giskard); NeMo Agent Toolkit: Making Agents Reliable (Nvidia).
• Tier 4 — Supporting skills. Getting Structured LLM Output (DotTxt); Pydantic for LLM Workflows (DeepLearning.AI); Building and Evaluating Data Agents (Snowflake).
• Tier 5 — Optional coding agents. Claude Code (Anthropic); Spec-Driven Development with Coding Agents (JetBrains).

Marquee names, sensible progression, the risk material promoted to its own tier. I very nearly hit “enroll” on all of it. And then the part of my brain that reviews models for a living cleared its throat and asked the question it always asks: how do you know this is any good?

The validation: interrogating my own reading list

The uncomfortable thing about a course list is that it markets itself. Every card looks equally current, because the thumbnails are equally glossy. But in applied AI, currency is everything. Frameworks ship breaking changes on a schedule that would make a fixed-income desk weep. A course that teaches you to drive a specific library the way it worked eighteen months ago is not a shortcut; it’s a trap with a completion certificate.

So I did the boring, load-bearing thing. I checked the vintage of every course before committing a single evening to it.

Getting the dates was itself a small exercise in not trusting the surface. The course cards don’t advertise a release date — you see a title, a duration, a difficulty, and nothing about age. But the platform is a Next.js app, and Next.js ships a page’s data to the browser as a blob of JSON embedded in the HTML (the __NEXT_DATA__ script tag). Buried in that blob, each course carried a quiet little released_at field that the interface never bothers to render. A few lines in the browser console were enough to pull it out:

const data = JSON.parse(document.getElementById('__NEXT_DATA__').textContent);
// walk the embedded query cache and collect { slug, name, released_at }
// ...then sort by date and stop kidding yourself about what's current

Five minutes of squinting at JSON told me more about the syllabus’s health than five hours of reading course descriptions would have. This is, more or less, my entire job compressed into a paragraph: the polished front-end is not the model; go find the data the front-end is quietly sitting on.

Then I split the courses into two buckets, because “old” isn’t one thing:

• Concept-durable courses teach mental models that don’t expire — what a transformer is, how to think like an attacker, why you evaluate before you deploy. A two-year-old course on those still earns its seat.
• API-volatile courses teach you to drive a specific, fast-moving library. Those rot. A mid-2024 course on an agent framework will hand you notebooks that no longer run against today’s version of that same framework.

Here is what validating the dates turned up (ages as of writing, mid-2026):

The perfect list, it turned out, had five soft spots — all clustered, predictably, in the hands-on agent-building tier where the frameworks churn fastest.

Draft two: the swaps

Where a stale, API-volatile course had a fresher replacement teaching the same thing, I swapped it. Where it didn’t, I kept it and made peace with rewriting a few code cells. Where a fresher course elsewhere already covered the ground, I cut the duplicate.

Two honest caveats. First, DSPy is not a like-for-like replacement for LangGraph — it’s a different paradigm (optimizing agents rather than wiring them as graphs). I chose it for freshness and for the MLflow angle; if you want to stay in the LangGraph world, its newest touchpoint is Long-Term Agentic Memory with LangGraph (Mar 2025). Second, the swaps cost time. The 2024 courses were bite-sized; their current replacements are comprehensive. My honest estimate for Tiers 1–4 roughly doubled, from ~30 hours to closer to 65. A season of evenings, not a long weekend. I decided current-and-longer beats dated-and-quick — which is, when you think about it, exactly the trade a risk function exists to make.

Where I am so far

I’ve finished Tier 1’s Agentic AI, and, setting aside the professional obligation to sound measured, I came out of it genuinely energized.

The reframe that landed hardest was almost embarrassingly simple. I’d been thinking of an LLM as a very articulate oracle: you ask, it answers, you move on. The course reoriented me toward agents as iterative, multi-step workflows that take action: plan, act, observe, revise, repeat. Once that clicked, my skepticism sharpened into something more useful. An oracle you can spot-check. A system that plans and acts across many steps, calling tools and touching real state along the way, needs the full apparatus of model risk: evaluation, monitoring, controls, and someone whose job it is to ask how it breaks.

Which is exactly why the tiers ahead are the ones I’m most impatient to reach. The evaluation and governance material isn’t the part I’ll dutifully endure to get certified. It’s the part where my existing instincts finally have somewhere to go. I suspect I’ll be an insufferable student in those modules, and I’m oddly looking forward to it.

A disclaimer, offered without apology

In the spirit of the model-risk documentation I spend my life reading and writing, one honest note on provenance. This whole exercise — the five-tier structure, the reordering that promotes evaluation and governance, and the recency validation that pulled every release date and swapped out the stale courses — came out of an extended back-and-forth with Claude, Anthropic’s AI, acting as co-author and thinking partner. I brought the constraints, the skepticism, and the risk lens; it drafted the first list, and then, when I muttered that I didn’t trust how old these things were, it was the one that went and dug the released_at dates out of the page and helped me sort the keepers from the fossils.

There’s a pleasing symmetry to that. I used an AI to design my plan for understanding AI, validated its first draft the way I’d validate any model handed to me, and then told you so plainly rather than pretending the syllabus sprang fully formed from my own head. If I’m going to spend the next several weeks learning to evaluate these systems, I might as well start by being transparent about working with one. Consider it the first entry in the validation report.