AI Agent Papers2024

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Tests whether agents can finish 369 tasks in real desktop applications, using executable checks of the final state. The original study found a wide gap between agent and human success rates.

Visit Website

Introduction

A convincing action sequence can still leave the wrong spreadsheet or browser state. Evaluating the resulting files and application state makes completion measurable without requiring an agent to follow one reference trajectory.

Key Findings

The 369-task Ubuntu suite spans desktop software, files, and multi-application workflows, testing more than browser navigation.
In the original experiments, the best agent reached 12.24% success versus 72.36% for humans; these are historical baselines.
Failures in screen grounding and application knowledge show why stronger text reasoning alone may not fix desktop control.

Who Should Read It

Great fit if you study computer-use agents and need reproducible, outcome-based evaluation. Look elsewhere if you need a lightweight text-only test: virtual machines and application-specific evaluators add setup costs, and the published benchmark focuses on Ubuntu.

Back

Information

Websitearxiv.org
OrganizationsThe University of Hong Kong, Carnegie Mellon University, Salesforce Research, University of Waterloo
AuthorsTianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei …
Published date2024/04/11

More Items

AI Agent Papers2026

AgentIR: Reasoning-Aware Retrieval for Deep Research Agents

Zijian Chen, Xueguang Ma +4University of Waterloo, University of Queensland +1

Embeds an agent's reasoning together with its search query to improve retrieval. Trains a 4B embedding model on synthesized deep-research trajectories, helping preserve intent across multi-step searches.

paper ai-agent retrieval embeddings reasoning

AI Agent Papers2025

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

Rulin Shao, Akari Asai +19University of Washington, Allen Institute for AI +4

Trains an 8B model to search, synthesize evidence, and write cited research reports. Its reinforcement-learning rewards use evolving rubrics grounded in retrieved evidence and contrasting model responses.

paper ai-agent rl research web-search+1

AI Agent Papers2025

Tongyi DeepResearch Technical Report

Baixuan Li, Bo Zhang +54Tongyi Lab, Alibaba Group

Trains long-horizon web researchers by combining agentic mid-training, synthetic trajectories, and reinforcement learning. A 30.5B-parameter model activates 3.3B parameters per token and learns to plan, search, and synthesize evidence.

paper ai-agent LLM RL research+4