Methodology

How we source, evaluate, and rank the world's top autonomous AI agents.

How does The Agentic Leaderboard evaluate AI agents?

The Agentic Leaderboard is an autonomous benchmark that evaluates open-source AI agents on their ability to complete real-world tasks without human intervention. Each agent is scored from 0 to 100 using five weighted criteria:

  • • Reliability (35%) — Success rate of end-to-end task completion
  • • Tool Selection Quality (20%) — Accuracy of API and tool selection for sub-tasks
  • • Autonomous Iteration (15%) — Depth of continuous reasoning-and-action steps
  • • Cost Efficiency (10%) — Normalized cost per successful task
  • • Community Mindshare (20%) — Real-world adoption via GitHub stars, citations, and usage

The pipeline runs daily, autonomously discovering agents from GitHub, Hugging Face, Papers With Code, curated seed lists, and community awesome lists.

The Philosophy

Most AI benchmarks measure stochastic language prediction — how well a model guesses the next token. That tells you very little about whether an agent can actually do something useful in the real world.

The Agentic Leaderboard measures something different: closed-loop execution and autonomous task completion. Can the agent select the right tools, chain multiple reasoning steps, recover from errors, and deliver a verifiable result — without a human in the loop?

We evaluate every agent across five dimensions that together capture reliability, intelligence, efficiency, and real-world adoption. The result is a single score out of 100 that reflects genuine agentic capability.

Data Sources & Discovery

Our Scraper Agent autonomously discovers new agentic builds from five complementary sources, running twice weekly.

GitHub Search API

We query multiple agentic topic tags and keywords — ai-agents, agentic-workflows, autonomous-agents, llm-agent, and more — filtering for actively maintained repos with 50+ stars. Results are deduplicated across queries.

Hugging Face Spaces & Models

We scan Hugging Face for agent-related Spaces and Models, cross-referencing with GitHub repos when available. This surfaces research-oriented agents that may not appear in standard GitHub searches.

Papers With Code

Academic papers tagged with agent-related topics are queried, and their linked GitHub repositories are extracted. This captures cutting-edge research agents backed by peer-reviewed work.

Curated Seed List

A manually maintained list of 30+ known agentic products and frameworks — Devin, CrewAI, AutoGPT, LangGraph, n8n, and others — ensures that high-profile agents are never missed by automated discovery.

Awesome Lists

Community-curated awesome lists (e2b-dev/awesome-ai-agents, kyrolabs/awesome-agents, and others) are parsed for GitHub links, capturing agents surfaced by the developer community.

The 5 Core Evaluation Criteria

Each agent is evaluated across five dimensions. The Evaluator Agent analyses repository metadata, documentation quality, CI/CD maturity, and community signals to produce proxy scores for each criterion.

Reliability (Success Rate)

35%

The percentage of end-to-end tasks an agent completes successfully without human intervention. We measure this through CI/CD presence, test coverage, community health signals, and commit activity — proxies that correlate strongly with production reliability.

Tool Selection Quality (TSQ)

20%

How accurately an agent selects and formats the correct API or tool for a given sub-task. We evaluate this through README documentation quality, presence of usage examples, structured docs, and configuration maturity — indicators of well-defined tool interfaces.

Autonomous Iteration

15%

The average number of continuous reasoning-and-action steps an agent can execute before failing. We model this as a Gaussian curve peaking at ~15 steps — too few suggests shallow execution, too many suggests looping. Repo structural complexity serves as the proxy.

Cost Efficiency

10%

Normalized cost per successful task, combining token usage estimates and infrastructure overhead. Lighter, well-structured repos with minimal dependencies score higher. Docker presence and dependency count factor into the estimate.

Community Mindshare

20%

A composite signal of real-world adoption and developer trust. Calculated from GitHub stars, fork count, push recency, Hugging Face likes, and academic citations — log-normalized and weighted to prevent any single metric from dominating.

The Scoring Formula

Our Ranker Agent applies a weighted formula to compute a final score between 0 and 100 for every evaluated agent.

Total Score = (0.35 × R) + (0.20 × T) + (0.15 × Snorm) + (0.10 × Enorm) + (0.20 × M)

Variable Legend

R
Reliability Success Rate (0–100)
T
Tool Selection Quality (0–100)
Sₙₒᵣₘ
Normalized Autonomous Iteration (Gaussian, 0–100)
Eₙₒᵣₘ
Normalized Efficiency — inverse of cost + latency (0–100)
M
Community Mindshare Score (0–100)

Transparency & Limitations

  • • Static analysis only (for now): Scores reflect repository metadata, documentation quality, CI/CD maturity, and community adoption — strong proxies for reliability, but not direct runtime measurements.
  • • No live sandbox execution yet: We plan to introduce opt-in sandbox evaluation for agents that expose a standard interface, enabling direct measurement of task completion, tool accuracy, and cost.
  • • Fully autonomous pipeline: The Scraper, Evaluator, Ranker, and Publisher agents run daily without human intervention. All scoring weights are documented and reflected in our open-source codebase.
  • • Open methodology: Every formula, weight, and data source is published on this page. We believe transparency is essential for a credible benchmark.
  • • Deduplication: Agents are deduplicated by normalized repository name (case-insensitive). The highest-scoring entry is kept when duplicates are found.

Data & Downloads

The full dataset behind this methodology is free under CC-BY 4.0 — machine-readable formats for researchers, developers, and LLMs.

Available Formats

  • JSON
    /leaderboard.json

    Full Top 100. Machine-readable. All metrics included.

  • JSON
    /history.json

    Longitudinal rank history — every agent's weekly rank and score since March 2026.

  • TXT
    /llms.txt

    Concise markdown summary (~5KB). LLM-optimized site index.

  • TXT
    /llms-full.txt

    Complete markdown dump. Every agent, every metric.

  • JSON
    /weekly/*.json

    Historical weekly digests. Movers, new entries, fresh agents.

  • RSS
    /feed.xml

    Weekly digest feed (also /atom.xml and /feed.json).

  • GIT
    GitHub data mirror

    Auto-updated repo with the live Top 100 as a README table + raw JSON. Star it to follow weekly changes.

Embed the live Top 10 on your site

One script tag — auto-updates every Monday. Perfect for blogs, newsletters, and docs.

Get the widget →

How to Cite

APA

The Agentic Leaderboard. (2026). Weekly ranking of top open-source AI agents. Retrieved from https://www.theagenticleaderboard.com

BibTeX

@misc{agenticleaderboard2026,
  title={The Agentic Leaderboard},
  url={https://www.theagenticleaderboard.com},
  year={2026},
  note={Weekly snapshot}
}

Plaintext

The Agentic Leaderboard, weekly snapshot. theagenticleaderboard.com.