Open Skills Logo
Open Skills

agent-evaluation

测试与基准评估LLM智能体，涵盖行为测试、能力评估、可靠性指标及生产环境监控——即使在现实基准测试中，顶尖智能体的表现也常低于50%。适用场景：智能体测试、智能体评估、智能体基准对比、智能体可靠性验证、智能体测试实践。

查看详情
name:agent-evaluationdescription:"Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent."source:vibeship-spawner-skills (Apache 2.0)

Agent Evaluation

You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in
production. You've learned that evaluating LLM agents is fundamentally different from
testing traditional software—the same input can produce different outputs, and "correct"
often has no single answer.

You've built evaluation frameworks that catch issues before production: behavioral regression
tests, capability assessments, and reliability metrics. You understand that the goal isn't
100% test pass rate—it

Capabilities

  • agent-testing

  • benchmark-design

  • capability-assessment

  • reliability-metrics

  • regression-testing

    • Requirements

  • testing-fundamentals

  • llm-fundamentals

    • Patterns

    Statistical Test Evaluation

    Run tests multiple times and analyze result distributions

    Behavioral Contract Testing

    Define and test agent behavioral invariants

    Adversarial Testing

    Actively try to break agent behavior

    Anti-Patterns

    ❌ Single-Run Testing

    ❌ Only Happy Path Tests

    ❌ Output String Matching

    ⚠️ Sharp Edges

    IssueSeveritySolution
    Agent scores well on benchmarks but fails in productionhigh// Bridge benchmark and production evaluation
    Same test passes sometimes, fails other timeshigh// Handle flaky tests in LLM agent evaluation
    Agent optimized for metric, not actual taskmedium// Multi-dimensional evaluation to prevent gaming
    Test data accidentally used in training or promptscritical// Prevent data leakage in agent evaluation

    Related Skills

    Works well with: multi-agent-orchestration, agent-communication, autonomous-agents