Tag
Explore by tags
A multi-task English NLU benchmark for evaluating models across nine tasks (acceptability, sentiment, paraphrase, similarity, and various NLI setups), with a diagnostic evaluation set and an online leaderboard to compare generalization and transfer learning.
Showed that fine-tuning a GPT model on public GitHub code yields a capable program synthesizer, and introduced HumanEval — the docstring-to-function benchmark that still anchors code-generation evaluation. A production variant powers GitHub Copilot.
Tests whether browser agents complete 812 tasks across realistic shopping, forum, development, and content-management sites. Evaluates final outcomes rather than matching a reference sequence of clicks.
Benchmark for evaluating general AI assistants with 466 short, real-world questions that require tool use, multimodality and reasoning; provides a public dev set and a withheld test set used for leaderboard evaluation.
A multiple-choice benchmark for evaluating LLM understanding in Traditional Chinese across 66 subjects (elementary to professional). Contains ~22K verified questions covering STEM, humanities, social sciences and Taiwan-specific topics, with standardized splits and model leaderboards under an MIT license.
Tests whether web agents can use visual evidence to finish 910 tasks across shopping, classifieds, and social forums. Execution-based checks distinguish successful actions from plausible explanations.
Explains when language models benefit from external programs, comparing tool selection, use, and creation. Evaluates performance gains against compute costs and identifies gaps in tool-use benchmarks.
Tests whether agents can finish 369 tasks in real desktop applications, using executable checks of the final state. The original study found a wide gap between agent and human success rates.
Provides manually curated Japanese instruction pairs (questions and safe reference answers) for improving LLM output safety, covering broad harm categories and regionally sensitive cases. Includes English meta-tags and standard splits for benchmarking and fine-tuning.
A 12K-question multiple-choice benchmark that raises difficulty by expanding choices to ten and adding reasoning-focused problems. Designed for robust LLM evaluation with cross-domain coverage, expert curation, and emphasis on chain-of-thought performance.
Orchestrates configurable deep-research agent workflows that combine LLMs, web search, and MCP tools to produce structured research reports and evaluation outputs. Supports LangGraph Studio, multiple model providers (OpenAI, Anthropic, local models), and Deep Research Bench evaluation for benchmarked comparisons.
A fine-grained, domain-labeled web corpus for NLP and LLM research that decomposes a deduplicated web crawl into 67 domains via iterative URL/content labeling and coarse→fine recall; includes per-domain statistics, validation subsets, and domain-benchmark analysis tools.