agent-orchestration-advisor
Design multi-agent AI workflows with clear boundaries, handoffs, and monitoring. Use when a complex PM task should run as parallel specialized agents instead of one linear process.
Author
Category
Product DesignInstall
Download and extract to your skills directory
Copy command and send to AI Agent for auto-install:
Purpose
Guide product managers through designing multi-agent workflows—breaking complex, repetitive PM tasks into parallel, specialized AI agents rather than linear, sequential processes or manual execution. Use this to transition from "document-heavy administrator" to "systems-level orchestrator" who coordinates a "living system" of AI agents, human teams, and market data interacting continuously.
Key Shift: From linear project management (one task at a time) to orchestration (multiple agents working simultaneously, each with clear boundaries and handoffs).
This is not about prompt writing—it's about architecting workflows where AI agents handle repetitive research, synthesis, and validation while PMs focus on strategy and decision-making.
Input
Works best with: The workflow or recurring task you want to orchestrate — described in a sentence or two, however manual or messy it is today.
Also useful: Where it breaks down now (too slow, too sequential, too dependent on you), the tools your team already uses, and whether you've worked through context-engineering-advisor first (it's the prerequisite discipline).
Anything supplied with the invocation itself — text after the skill name, a pasted context dump, or an appended ARGUMENTS: line — counts as answers already given. Use it and skip whatever it covers; don't re-ask.
Arriving empty-handed? That works too. The advisor opens by asking which PM workflow eats the most of your week, then walks the four orchestration dimensions against it.
Example invocation: Design an orchestration for our weekly competitive intel: today one PM spends 6 hours scraping, summarizing, and briefing — sequentially.
Key Concepts
Orchestration vs. Project Management
| Dimension | Project Management | Orchestration |
|---|---|---|
| Approach | Linear oversight of schedules and human tasks | Managing "living system" where AI agents, humans, and data interact continuously |
| Task Flow | Sequential (finish A, then B, then C) | Parallel (A, B, C run simultaneously) |
| PM Role | Document-heavy administrator | Systems-level leader coordinating automated systems + human judgment |
| Focus | Output (features shipped) | Outcome (business results, learning velocity) |
| Risk Management | Manual tracking and mitigation | Real-time monitoring with agentic systems flagging gaps |
Critical Insight: Orchestration is not about replacing humans—it's about force-multiplying human judgment by automating repetitive, time-consuming tasks.
The Four Dimensions of Orchestration
1. Coordination of Multi-Agent Workflows
Breaking complex tasks into specialized agents that run in parallel.
Example:
- Agent A: Competitive intel (research agent)
- Agent B: Customer synthesis (synthesis agent)
- Agent C: Roadmap gap analysis (analysis agent)
- Total time: 8 hours (limited by slowest agent), PM reviews outputs in 2 hours = 10 hours total, 5 hours saved
Key Principle: Shift from manual selection to hypothesis orchestration—agents generate hypotheses, PM validates and decides.
2. Leadership of Cross-Functional AI Pods
Governing diverse teams (data scientists, ML engineers, compliance, ethicists) to ensure solutions are scalable, ethical, and aligned.
What it includes:
PM Role: Guardian of Governance—ensures AI systems reflect company values.
3. Launch Control Tower Function
Real-time monitoring of organizational readiness across functions using agentic systems to flag gaps before critical failures.
What it monitors:
Key Principle: Agentic systems act as early warning system—flag gaps before they become blockers.
4. Strategic Intent Alignment (Context Engineering Applied)
Feeding AI agents the correct mix of mission, constraints, and priorities to ensure automated decisions reflect company values.
Connection: This is context engineering at the orchestration layer. See context-engineering-advisor for foundations.
What agents need:
The Four AI Management Workflows (Productside Blueprint)
Every PM must master these workflows to move fast while staying grounded:
- Create AI workspace that remembers product domain, research, JTBD, personas, constraints
- Skill:
context-engineering-advisor- Automated validation tests for AI reasoning
- Generate synthetic data, run workflows against traces
- Eliminates 80% of hallucination risk
- Agents handle repetitive tasks (competitive intel, customer synthesis, roadmap gaps)
- PM focuses on strategy
- Generate clickable prototypes from context workspace
- Collapse feedback loops from weeks to hours
- Connection:
pol-probe-advisor (Vibe-Coded PoL Probes)AI-Shaped Problems (Teresa Torres)
What makes a problem "AI-shaped"?
Key Insight: "While AI makes building easier, choosing what to build remains the primary challenge." Orchestration helps with the "building" part so PMs can focus on "choosing."
The Four Big Risks (Marty Cagan, AI Era)
The orchestrator manages these risks across the organization:
| Risk | Standard Definition | AI Era Layer | Orchestrator's Role |
|---|---|---|---|
| Value Risk | Will customers pay? | Does AI provide enough incremental benefit vs. cost? | Validate value with PoL probes before orchestrating |
| Usability Risk | Can users figure it out? | Is the UX right? (Chat often isn't!) | Test workflows with real users |
| Feasibility Risk | Can we build it? | Do we understand "physics of AI"? Token budget? | Design within technical constraints |
| Viability Risk | Can we sell/support? | Legal limits? Data privacy? Sustainable OpEx? | Monitor costs, compliance, scalability |
Anti-Patterns (What This Is NOT)
When to Use This Skill
✅ Use this when:
❌ Don't use this when:
context-engineering-advisor)Application
This interactive skill uses adaptive questioning to design multi-agent workflows step-by-step.
Step 0: Gather Context
Agent asks:
Before we design your orchestrated workflow, let's gather context:
Current PM Work:
Desired Outcome:
Technical Context:
context-engineering-advisor)You can describe briefly or paste examples.
Step 1: Assess if Task is AI-Shaped
Agent asks:
Let's determine if your task is suited for orchestration. Which statement best describes your situation?
Select one:
- Examples: Compiling competitive intelligence, synthesizing customer feedback, tracking feature requests
- Assessment: High potential for orchestration
- Examples: Research competitors → Synthesize findings → Identify gaps → Update roadmap (15 hours total, done sequentially)
- Assessment: Good candidate for parallelization
- Examples: Risk analysis for 50 features, synthesizing 100 user interviews, tracking 20 competitors
- Assessment: Perfect for orchestration (agents ensure consistency)
- Examples: Stakeholder negotiation, team coaching, ethical trade-offs
- Assessment: Not a good candidate for orchestration (keep human-driven)
User response: [Selection]
Agent analyzes:
Based on your selection:
Step 2: Decompose Task into Agent-Specific Subtasks
Agent asks:
Let's break your task into discrete subtasks, each handled by a specialized agent.
For the task you selected, what are the distinct steps?
Example Decomposition (Competitive Intelligence):
Manual Process (Old):
Total: 12 hours, done sequentially
Orchestrated Process (New):
Total: Agents A-C run in parallel (3 hours), then D and E run sequentially (2 hours) = 5 hours + 1 hour PM review = 6 hours total (50% time saved)
Agent offers:
I'll help you decompose your task. Describe your current process step-by-step, and I'll identify:
User response: [Describe process]
Agent provides: Decomposed workflow with agent assignments.
Step 3: Design Parallel vs. Sequential Flows
Agent asks:
Now let's design the workflow topology—which agents run in parallel vs. sequentially.
Three Flow Patterns:
Pattern 1: Full Parallel (Fastest)
[Agent A] ─┐
[Agent B] ─┼─→ [PM Review] → [Decision]
[Agent C] ─┘Pattern 2: Pipeline (Sequential with Handoffs)
[Agent A] → [Agent B] → [Agent C] → [PM Review] → [Decision]Pattern 3: Hybrid (Parallel + Sequential)
[Agent A] ─┐
[Agent B] ─┼─→ [Agent D] → [Agent E] → [PM Review] → [Decision]
[Agent C] ─┘For your task, which pattern fits best?
Options:
User response: [Selection]
Agent provides: Visual workflow diagram with agent dependencies.
Step 4: Define Agent Boundaries and Handoffs
Agent asks:
Now let's define what each agent knows (context boundaries) and how agents hand off to each other.
For each agent, define:
- Always-available (from context engineering)
- Retrieved on-demand (specific to this run)
Example (Competitive Intelligence Orchestration):
Agent A: Competitor Research Agent
- Always-available: List of competitors (from constraints registry)
- Retrieved: Competitor URLs (from database)
{competitor, features[], last_updated}Agent D: Synthesis Agent
- Always-available: Report template, writing tone (from context workspace)
- Retrieved: Outputs from Agents A, B, C
Agent offers:
I'll help you define boundaries for each agent. For your workflow, let's define:
User response: [Describe agent needs]
Agent provides: Complete agent boundary definitions + handoff specifications.
Step 5: Implement Strategic Intent Alignment
Agent asks:
To prevent agents from "going rogue," we need to feed them the correct mix of mission, constraints, and priorities.
Strategic Intent Checklist (Per Agent):
- Example: "Identify competitive threats that could impact Q2 roadmap"
- Example: "Do not recommend features outside our strategic focus areas"
- Example: "Do not scrape data from sites that prohibit it (check robots.txt)"
- Example: "Prioritize features launched in last 30 days over older features"
- Example: "Only cite information from official product pages, release notes, or verified blog posts"
Connection: This is context engineering applied to orchestration. See context-engineering-advisor for deeper guidance.
Agent offers:
For each agent in your workflow, let's define:
User response: [Define strategic intent per agent]
Agent provides: Strategic intent document for each agent.
Step 6: Set Up Launch Control Tower Monitoring
Agent asks:
The Launch Control Tower provides real-time monitoring to flag gaps before they become critical failures.
What to Monitor (Three Dimensions):
1. Agent Performance:
2. Organizational Readiness (If Launching a Feature):
3. Risk Flags:
Monitoring Approach:
Option 1: Manual Dashboard
Option 2: Automated Monitoring
Option 3: Hybrid
Which monitoring approach fits your team's maturity?
Options:
User response: [Selection]
Agent provides: Monitoring plan with specific metrics and alert thresholds.
Step 7: Create Evaluation Plan
Agent asks:
To ensure orchestrated workflows produce high-quality outputs, we need an evaluation framework.
The Four Eval Types (From Research):
| Eval Type | Method | When to Use |
|---|---|---|
| Golden Datasets | 20-100 examples of real inputs + expected outputs | Baseline quality measurement |
| Code Assertions | Rules output must follow (e.g., must contain valid JSON) | Structural validation |
| LLM-as-Judge | Powerful model (GPT-4o) grades agent output | Rapid automated feedback |
| Human Evals | PM manually reviews traces | Ultimate check for "taste" and "product sense" |
Evaluation Process:
For your workflow, which evals make sense?
Options:
User response: [Selection]
Agent provides: Evaluation plan with specific eval types, frequencies, and success criteria.
Step 8: Implementation Plan & Next Steps
Agent synthesizes:
Here's your complete orchestrated workflow plan:
┌─────────────────────────────────────────────────────────────┐
│ ORCHESTRATED WORKFLOW: [Your Task Name] │
├─────────────────────────────────────────────────────────────┤
│ │
│ TOPOLOGY: [Full Parallel / Pipeline / Hybrid] │
│ │
│ AGENTS: │
│ • Agent A: [Name] - [Purpose] │
│ • Agent B: [Name] - [Purpose] │
│ • Agent C: [Name] - [Purpose] │
│ │
│ FLOW: │
│ [Agent A] ─┐ │
│ [Agent B] ─┼─→ [Agent D] → [PM Review] → [Decision] │
│ [Agent C] ─┘ │
│ │
│ CONTEXT BOUNDARIES: │
│ • Always-available: [List] │
│ • Retrieved on-demand: [List] │
│ │
│ STRATEGIC INTENT: │
│ • Mission: [Per agent] │
│ • Constraints: [Per agent] │
│ • Priorities: [Per agent] │
│ │
│ MONITORING: │
│ • Approach: [Manual / Automated / Hybrid] │
│ • Metrics: [List] │
│ • Alert thresholds: [List] │
│ │
│ EVALUATION: │
│ • Golden Datasets: [Y/N] │
│ • Code Assertions: [Y/N] │
│ • Human Evals: [Frequency] │
│ │
│ TIME SAVINGS: │
│ • Manual (old): [X hours] │
│ • Orchestrated (new): [Y hours] │
│ • Savings: [X-Y hours per week] │
└─────────────────────────────────────────────────────────────┘Implementation Roadmap:
Week 1: Build Context Foundations
context-engineering-advisorWeek 2: Implement First Agent
Week 3: Add Remaining Agents
Week 4: Set Up Monitoring & Evals
Week 5+: Iterate & Scale
Success Criteria:
Agent offers:
Would you like me to:
Examples
Example 1: Competitive Intelligence Orchestration
Context:
Goal: Reduce to 6 hours with orchestrated workflow
Workflow Design:
Agents:
Topology: Hybrid (A, B, C in parallel → D → E → PM Review)
Strategic Intent:
Monitoring:
Evaluation:
Result:
Example 2: Customer Feedback Synthesis
Context:
Goal: Reduce to 2 hours with orchestrated workflow
Workflow Design:
Agents:
Topology: Pipeline (A → B → C → D → E → F → PM Review)
Strategic Intent:
Monitoring:
Evaluation:
Result:
Example 3: Risk Analysis at Scale
Context:
Goal: Reduce to 5 hours with orchestrated workflow
Workflow Design:
Agents:
Topology: Full Parallel (A, B, C, D run simultaneously → E → PM Review)
Strategic Intent:
Monitoring:
Evaluation:
Result:
Common Pitfalls
1. Orchestrating Before Context Engineering
Failure Mode: Building agent workflows without context foundations (constraints, glossary, evidence standards).
Consequence: Agents produce inconsistent outputs, violate constraints, hallucinate.
Fix: Complete context-engineering-advisor first. Build constraints registry, operational glossary, strategic intent documents.
2. Over-Orchestrating Simple Tasks
Failure Mode: Creating complex multi-agent workflows for tasks that take <2 hours per week.
Consequence: Orchestration overhead (setup, monitoring, maintenance) exceeds time saved.
Fix: Only orchestrate tasks that take 5+ hours per week or require consistency at scale.
3. No Evaluation Plan
Failure Mode: "Set it and forget it"—agents run without quality checks.
Consequence: Quality drift over time, unnoticed hallucinations, constraint violations.
Fix: Implement Golden Datasets + weekly Human Evals at minimum. Build failure mode taxonomy, create automated evals.
4. Ignoring Handoff Failures
Failure Mode: Assuming agents will correctly pass data to each other without testing.
Consequence: Agent B receives malformed data from Agent A, produces garbage output.
Fix: Test handoffs explicitly. Validate data format at each handoff. Use Code Assertions to enforce structure.
5. Forgetting the "PM Review" Step
Failure Mode: Treating orchestrated workflows as fully autonomous—no human oversight.
Consequence: Agents make decisions that lack context, empathy, or strategic alignment.
Fix: Always include PM Review step. Agents generate hypotheses/recommendations; PM validates and decides.