AI Dataset2018
Alex Wang, Amanpreet Singh +4New York University, Paul G. Allen School of Computer Science & Engineering, University of Washington +1
A multi-task English NLU benchmark for evaluating models across nine tasks (acceptability, sentiment, paraphrase, similarity, and various NLI setups), with a diagnostic evaluation set and an online leaderboard to compare generalization and transfer learning.
Showed that fine-tuning a GPT model on public GitHub code yields a capable program synthesizer, and introduced HumanEval — the docstring-to-function benchmark that still anchors code-generation evaluation. A production variant powers GitHub Copilot.
AI Dataset2024
Hisami Suzuki, Satoru Katsumata +4LLM-jp (Center for Large Language Model Research and Development, National Institute of Informatics), National Institute of Informatics
Provides manually curated Japanese instruction pairs (questions and safe reference answers) for improving LLM output safety, covering broad harm categories and regionally sensitive cases. Includes English meta-tags and standard splits for benchmarking and fine-tuning.
AI Agent2024
Orchestrates configurable deep-research agent workflows that combine LLMs, web search, and MCP tools to produce structured research reports and evaluation outputs. Supports LangGraph Studio, multiple model providers (OpenAI, Anthropic, local models), and Deep Research Bench evaluation for benchmarked comparisons.
AI Infra2024
Full-stack AI red‑teaming platform that fingerprints AI infrastructure for known CVEs, audits MCP servers and agent skills with LLM-driven analysis, and runs cross-model jailbreak evaluations; designed for hands-on security assessment of AI deployments.
AI Dataset2025
A benchmark dataset for evaluating MLLM-driven interactive webpage code generation: provides prototyping screenshots, action.json interaction metadata, and example generation scripts across 127 webpages and 374 interactions to test dynamic UI-to-code capabilities.
AI Agent2025
Simulates adversarial attacks against LLMs and AI agents to surface vulnerabilities (e.g., jailbreaks, prompt injection, PII leakage) and ships guardrails to block risky inputs/outputs; runs locally and can be driven from CLI or Python.
AI Dataset2025
Jake Poznanski, Jon Borchardt +7Allen Institute for Artificial Intelligence (AI2), AllenNLP / olmOCR team
Benchmark for evaluating OCR systems that convert PDFs and scans into Markdown and structured text: 1,403 PDFs and 7,010 unit tests covering text presence/absence, reading order, tables, and math formula accuracy. Diverse sources and ODC-BY-1.0 license for research use.
AI Agent2025
Autonomously executes diverse biomedical research tasks by combining LLM reasoning, retrieval-augmented planning, and code-based execution. Includes a web UI and Gradio demo, a curated Know‑How library, MCP integration, and a biology-tailored reasoning model (Biomni‑R0).
AI Others2025
Argues AI has entered its 'second half': a working recipe (language pre-training priors + scale + reasoning) now generalizes RL across tasks, so the bottleneck shifts from inventing methods to defining problems and rethinking evaluation.
Practical, full-stack tutorial for building Retrieval-Augmented Generation (RAG) systems—covers data preprocessing, vector embedding and indexing, hybrid and multimodal retrieval, generation integration, evaluation and production-ready engineering. Includes hands-on projects and examples for developers with Python experience.
AI Agent2025
Evaluates and optimizes AI agents and language models in containerized environments, supporting large-scale parallel benchmarks and RL rollouts. Integrates with third‑party providers for thousands of parallel environments and serves as the official harness for Terminal‑Bench.