Tag

Crowd-sourced leaderboard that tracks token usage, popularity trends and market share of large language models on the OpenRouter gateway.

Objective benchmark leaderboard from the OpenCompass community, scoring LLMs and LVLMs across 100+ datasets in five capability dimensions.

Unified framework for few-shot evaluation of generative language models across 60+ academic benchmarks. Supports multiple model backends (Hugging Face, vLLM, APIs, local servers), configurable prompts/YAML configs, and reproducible exports for leaderboards and research comparisons.

Publishes real-time, human-voted leaderboards that rank frontier AI models across chat, code, image, video and more. Uses crowdsourced pairwise voting and Elo-style scores with per-arena breakdowns and historical data.

VLMEvalKit is an open-source evaluation toolkit for large vision-language models (VLMs/LVLMs). It enables one-command evaluation across many benchmarks, supports generation-based evaluation with optional LLM answer extraction, and provides leaderboards and reproducible pipelines.

Provides leaderboard-ready test splits for the Open ASR Leaderboard: converts unsafe custom loaders to Parquet, sorts samples by audio length, and packages eight ESB test sets (LibriSpeech, Common Voice, GigaSpeech, SPGISpeech, etc.) for reproducible ASR benchmarking.

A human-verified subset of 500 SWE-bench test cases for evaluating models that resolve GitHub issues into PRs using unit-test verification. Contains problem statements and base commits (pre-fix) for reproducible unit-test based evaluation; suitable for benchmarking code-fix and issue-resolution capabilities.

Benchmark dataset for evaluating agents on long-horizon software-engineering tasks (repo-level patches, test-driven fixes). Includes golden patches, related tests, and problem statements in parquet format; aimed at agent debugging and code-modification evaluation but requires full test environments.

A challenge repository for training the best language model that fits inside a 16,000,000‑byte (16MB) submission artifact; provides baseline training code, FineWeb bpb evaluation, a public leaderboard, and compute-grant instructions for short 8×H100 runs.

Benchmarks document-parsing systems on real-world enterprise PDFs and images—evaluates tables, charts, content faithfulness, semantic formatting, and visual grounding with human-verified, rule-level tests. Ships with ~2,000 pages, ~169K test rules, and an open evaluation framework for end-to-end pipeline scoring.

Benchmark dataset for evaluating clinician-facing chat assistants: physician-authored conversations plus rubric items, use-case and difficulty labels, specialty metadata, and a built-in canary to reduce benchmark contamination. Hosted on Hugging Face under an MIT license.

Evaluates LLM-driven agents on long-horizon, policy-rich U.S. healthcare workflows using 75 clinical task fixtures and a 20-app MCP simulator; includes task fixtures, shared worlds, and leaderboard integration (Managed-Care handbook is gated).