Methodology

How we source, evaluate, and rank the world's top autonomous AI agents.

How does The Agentic Leaderboard evaluate AI agents?

The Agentic Leaderboard is an autonomous benchmark that evaluates open-source AI agents on their ability to complete real-world tasks without human intervention. Each agent is scored from 0 to 100 using five weighted criteria:

  • • Reliability (35%) — Success rate of end-to-end task completion
  • • Tool Selection Quality (20%) — Accuracy of API and tool selection for sub-tasks
  • • Autonomous Iteration (15%) — Depth of continuous reasoning-and-action steps
  • • Cost Efficiency (10%) — Normalized cost per successful task
  • • Community Mindshare (20%) — Real-world adoption via GitHub stars, citations, and usage

The pipeline runs daily, autonomously discovering agents from GitHub, Hugging Face, Papers With Code, curated seed lists, and community awesome lists.

The Philosophy

Most AI benchmarks measure stochastic language prediction — how well a model guesses the next token. That tells you very little about whether an agent can actually do something useful in the real world.

The Agentic Leaderboard measures something different: closed-loop execution and autonomous task completion. Can the agent select the right tools, chain multiple reasoning steps, recover from errors, and deliver a verifiable result — without a human in the loop?

We evaluate every agent across five dimensions that together capture reliability, intelligence, efficiency, and real-world adoption. The result is a single score out of 100 that reflects genuine agentic capability.

Data Sources & Discovery

Our Scraper Agent autonomously discovers new agentic builds from five complementary sources, running twice weekly.

GitHub Search API

We query multiple agentic topic tags and keywords — ai-agents, agentic-workflows, autonomous-agents, llm-agent, and more — filtering for actively maintained repos with 50+ stars. Results are deduplicated across queries.

Hugging Face Spaces & Models

We scan Hugging Face for agent-related Spaces and Models, cross-referencing with GitHub repos when available. This surfaces research-oriented agents that may not appear in standard GitHub searches.

Papers With Code

Academic papers tagged with agent-related topics are queried, and their linked GitHub repositories are extracted. This captures cutting-edge research agents backed by peer-reviewed work.

Curated Seed List

A manually maintained list of 30+ known agentic products and frameworks — Devin, CrewAI, AutoGPT, LangGraph, n8n, and others — ensures that high-profile agents are never missed by automated discovery.

Awesome Lists

Community-curated awesome lists (e2b-dev/awesome-ai-agents, kyrolabs/awesome-agents, and others) are parsed for GitHub links, capturing agents surfaced by the developer community.

The 5 Core Evaluation Criteria

Each agent is evaluated across five dimensions. The Evaluator Agent analyses repository metadata, documentation quality, CI/CD maturity, and community signals to produce proxy scores for each criterion.

Reliability (Success Rate)

35%

The percentage of end-to-end tasks an agent completes successfully without human intervention. We measure this through CI/CD presence, test coverage, community health signals, and commit activity — proxies that correlate strongly with production reliability.

Tool Selection Quality (TSQ)

20%

How accurately an agent selects and formats the correct API or tool for a given sub-task. We evaluate this through README documentation quality, presence of usage examples, structured docs, and configuration maturity — indicators of well-defined tool interfaces.

Autonomous Iteration

15%

The average number of continuous reasoning-and-action steps an agent can execute before failing. We model this as a Gaussian curve peaking at ~15 steps — too few suggests shallow execution, too many suggests looping. Repo structural complexity serves as the proxy.

Cost Efficiency

10%

Normalized cost per successful task, combining token usage estimates and infrastructure overhead. Lighter, well-structured repos with minimal dependencies score higher. Docker presence and dependency count factor into the estimate.

Community Mindshare

20%

A composite signal of real-world adoption and developer trust. Calculated from GitHub stars, fork count, push recency, Hugging Face likes, and academic citations — log-normalized and weighted to prevent any single metric from dominating.

The Scoring Formula

Our Ranker Agent applies a weighted formula to compute a final score between 0 and 100 for every evaluated agent.

Total Score = (0.35 × R) + (0.20 × T) + (0.15 × Snorm) + (0.10 × Enorm) + (0.20 × M)

Variable Legend

R
Reliability Success Rate (0–100)
T
Tool Selection Quality (0–100)
Sₙₒᵣₘ
Normalized Autonomous Iteration (Gaussian, 0–100)
Eₙₒᵣₘ
Normalized Efficiency — inverse of cost + latency (0–100)
M
Community Mindshare Score (0–100)

Transparency & Limitations

  • • Static analysis only (for now): Scores reflect repository metadata, documentation quality, CI/CD maturity, and community adoption — strong proxies for reliability, but not direct runtime measurements.
  • • No live sandbox execution yet: We plan to introduce opt-in sandbox evaluation for agents that expose a standard interface, enabling direct measurement of task completion, tool accuracy, and cost.
  • • Fully autonomous pipeline: The Scraper, Evaluator, Ranker, and Publisher agents run daily without human intervention. All scoring weights are documented and reflected in our open-source codebase.
  • • Open methodology: Every formula, weight, and data source is published on this page. We believe transparency is essential for a credible benchmark.
  • • Deduplication: Agents are deduplicated by normalized repository name (case-insensitive). The highest-scoring entry is kept when duplicates are found.